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ABSTRACT 

School system leaders are discovering the power of data for 
promoting school improvement. This guide provides practitioners, parents, and 
community with insights and tools for cultivating a districtwide culture of 
data-driven inquiry. Chapter 1 points out that data provide quantifiable 
proof, taking the emotion and rancor out of the decision-making process. 
Chapter 2 recommends that districts use multiple assessment and nonassessment 
measures when assessing student performance. Because methods for data 
collection vary, superintendents should start by asking what they need to 
know and what data will help answer their questions. Chapter 3 recommends 
that teachers, principals, district staff, and the community should be 
involved in gathering, analyzing, and discussing data; and that crafting a 
comprehensive strategic communications plan can help superintendents identify 
key messages, audiences, results, and effective tools. Chapter 4 emphasizes 
that successful data-driven decision-making requires a shift in the culture 
of a school district that encourages the use and analysis of data without 
fear of reprisal. Appendix A contains accountability measures, Appendix B 
contains advice and techniques for working with the media, Appendix C is a 
list of resources, and Appendix is a glossary. (RT) 
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About AASA 

The American Association of School Administrators, founded in 1865, is the professional 
organization for over 14,000 educational leaders across the United States and in other 
countries. AASA's mission is to support and develop effective school system leaders who 
are dedicated to the highest quality public education for all children. 



'Foreword. 



School system leaders are discovering the power of data for promoting school improvement. 
With recent advances in technology and the increased demand for assessing student learning, 
an unprecedented amount of data are available to educators. School districts across America are 
beginning to use the tools necessary to make effective use of the data. In addition to test scores, 
many educators are collecting data about citizenship, character, healthy lifestyles, school climate 
and parental and community involvement. 



One superintendent reflected that "We spend a lot of time on testing but not much time on 
what to do with the test results." As educators shift their focus from simply reporting test results 
to using the data to improve instruction, data become essential ingredients in school improve- 
ment. Educators know that the effective use of data can measure student progress, evaluate 
program and instructional effectiveness, guide curriculum development and resource allocation, 
promote accountability and, most importantly, ensure that every child learns. 



Using Data to Improve Schools: What's Working is an easy-to-read guide to using data to drive 
school improvement. School system leaders and their staffs can learn from this book how to 
build a districtwide culture of inquiry that values the use of data for sound decision-making. 
School board members, parents and community members interested in helping improve schools 
will find tools for their work as well in this guide. It describes the challenges and the successes of 
educators from districts both large and small committed to using data. 



We are sure that you will find this guide useful in your ongoing efforts to provide leadership 
to your schools and communities. 

Paul D. Houston, Ph.D. 

Executive Director 

American Association of School Administrators 
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• What goals has your school district set for the next three years? 

• What data will help judge whether the district is meeting its goals? 

• How are superintendents using data they currently collect to improve 
student achievement over time? 

• What additional data must be collected and why? 

• In what ways are teachers , principals , district staff and the community 
involved in data collection and analysis? 

any superintendents have a powerful ally on their side: data. Increasingly, 



superintendents are using data to make smarter decisions, and they are getting results. 
Instead of responding defensively to critics, they are armed with facts and figures that tell a 
more complete story and help critics understand the root causes of the challenges schools face. 

"Data-driven decision-making is about gathering 
data to understand if a school or district is meeting its | 



purpose and vision," says Victoria Bernhardt, author I IfL the superintendent, cannot talk about how 

Of Data Analysis for Comprehensive Schoolwide j much learning has occurred , then I'm not achievir 

Improvement. (See A Closer Look on page 2.) "If we do i what I had hoped to as superintendent. 



Bernhardt says, superintendents should strive for ! 

"focused acts of improvement," which occur when 

schools are clear about their purpose, about what they expect students to know, and about what 
they expect students to be able to do. 

In data-driven districts, superintendents work side by side with other administrators, teachers, 
principals and parents to ensure all children achieve. Everyone strives toward common goals. 
Data provide quantifiable proof, taking the emotion and rancor out of what can be tough calls 
for superintendents and school boards (e.g., dismantling a popular but ineffective program or 
closing a school). Data also provide the substance for meaningful, ongoing dialogue within the 
educational community. 

For the past several years, Superintendent Gerrita Postlewait and the Horry County (S.C.) 
Public School Board have made data-driven decision-making the cornerstone of school 
improvement. "This is now a school system that focuses on results," Postlewait says. "If I, as the 




not have a target, we could make decisions that essen- 
tially lead to 'random acts of improvement/" Instead, 



— Gerrita Postlewait , superintendent, 
Horry County (S.C.) Public Schools 
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What is 

data-driven 

decision-making? 

• Collecting data 

• Analyzing data 

• Reporting data 

• Using data for school 
improvement 

• Communicating 
through data 



Data-driven decision- 
making requires a cultural 
shift in thinking that must 
be nurtured so ail stake- 
holders are committed to 
this effort. 



Data heip: 

Measure student progress 
Make sure students don't fall through the cracks 
Measure program effectiveness 
Assess instructional effectiveness 
Guide curriculum development 
Allocate resources wisely 
Promote accountability 
Report to the community 
Meet state and federal reporting requirements 
Maintain educational focus 
Show trends (but not necessarily solutions) 



superintendent, cannot talk about how much learning has occurred, then I'm not 
achieving what I had hoped to as superintendent." (See Getting Results on page 3.) 

Postlewait and other superintendents acknowledge that data were not always at 
the forefront of their decision-making. Until recently, they collected data only 
because gathering this information was mandated by the state in return for 
funding. Rarely were data used to analyze whether children were learning at 
grade level, teachers were using sound instructional practices or parents were 
pleased with the quality of schools. 

"Many school improvement plans are hit or miss," observes Superintendent 
Roland Smit, who heads the 600-student Mobridge (S.D.) School District. "We may 
think we should work on a specific area but we don't always look at the indicators 
that may tell us this is, in fact, something we should work on. With the help of 
data, we can zero in on our weaknesses." Data help superintendents like Smit 
make decisions with far greater precision and clarity. 

Data help district and school leaders craft a sound blueprint with measurable 
results for continuously improving schools so decisions are no longer based on 
incomplete or biased information. The challenge for superintendents is to know 
specifically what they are evaluating. For example, is a particular math program 

effective? Which students are showing the great- 
est gains and why? How can we target those 
who are showing the least improvement? 
Superintendents can focus on a specific grade 
level or measure overall progress toward meeting 
math standards, or they can use data to see how 
many students are at levels of proficiency or 
excellence in math. 



Data do not help: 

• If the data are not valid and reliable 

• If appropriate questions are not asked after reviewing 
the data 

• If data analysis is not used for making wise decisions 



A Closer Look 

Victoria Bernhardt 

Author, Data Analysis for Comprehensive 



Schoolwide Improvement 

For many data are confusing, even intimidating, but 
in your book you say data are logical if we think 
about what we need to know and why. Tell us more 
about that 

If questions can be created, the data that are 
needed to answer the questions are very logical. 
For example, for the question "How well are we 
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doing?" one would probably want to 
look at student achievement results 
on standardized tests and state or dis- 
trict assessments to get an idea of 
how students in the district are scor- 
ing right now. For the question "Are 
all students learning?" one might 
want to take the general student 
achievement analysis to a deeper 
level, looking at the distribution of 
scores to understand which students 
are scoring below mastery, and how 
far they are scoring below mastery. 

You also write that data can help uncov- 
er solutions to some of the toughest chal- 
lenges schools face today. How so? 

I think data can help us see things 
we might not see otherwise. We 
might have processes or programs in 
operation for years. Once we look at 
the data from all angles, we might 
find that a program is not helping all 
students learn. Data help us get to 
the root causes of a problem so we 
solve the problem and not just the 
symptom. 

In what ways can superintendents use 
data to improve student achievement? 

Student achievement data, for exam- 
ple, can help superintendents under- 
stand which instructional strategies 
are creating the best results and see 
where additional training might be 
needed. Perceptions data can tell 
superintendents about parent, stu- 
dent and staff satisfaction with the 
learning environment, which also 
could reveal areas in need of 
improvement. Staff perceptions can 
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Getting Results 

Horry County S.C. 

In the early 1990s, the Horry County (S.C.) School Board was 
granted the authority to raise taxes. School board members, 
parents and community members in the racially and economi- 
cally diverse school district of 29,000 students wanted to know 
the district was spending their taxpayer dollars wisely. 

Cerrita Postlewait, who was an instructional specialist with 
the district at the time, says the district staff began by re- 
examining their expectations. "For example, what kind of 
achievement results did we want to see/' Postlewait says. 
"What kind of activities did we want? Those questions and 
others kick-started the data-driven decision-making process." 

District leaders, principals and teachers learned a lot along 
the way, including that at-risk four-year-olds enrolled in the 
district's structured academic prekindergarten program scored 
consistently higher in reading, math and writing in first, 
second and third grades than students who participated in 
preschool programs sponsored by other governmental agen- 
cies. In fact, by third grade, 75 percent of the students who 
attended the district's preschool program scored higher than 
their counterparts in reading and writing. 

After collecting and analyzing the data, the district 
expanded the prekindergarten program by an additional 200 
children. "As we prepared our budget, we had a way to talk 
with our public about why this proposed change was impor- 
tant," Postlewait says. "We could show a reduction in the 
amount of time we have to spend with these students in reme- 
dial education a little later in grades one, two and three." 

Postlewait became superintendent of the district in 1997. 
"It's chaotic and bone-wearying work, but it's worth it because 
we are now seeing gains in student achievement. This is the 
first year we have had results to celebrate." 
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tell superintendents what is possible. Demographic data can provide valuable information about 
meeting the learning needs of students in the future, including: How might the makeup and size 
of the school population change? Or, how many new teachers with specialties will be needed in 
the near future? 

If you are a superintendent using data for the first time, where do you begin? How do you get started? 

I would start by understanding what data exist in the district 
and see what data I can get easily elsewhere, like the state edu- 
cation department or other sources. Undoubtedly, I could get 
demographic data, such as enrollment and attendance figures. 

I also believe that perceptions data is very important for under- 
standing schools and the health of the district. I would admin- 
ister online questionnaires to students, staff and parents. Then, 
if I were really on the ball with the data analysis, I would set up 
a comprehensive database to store and analyze the data at the district, school, classroom, 
teacher and student levels. 

What sort of leadership must a superintendent provide as more and more teachers, district staff, board 
members and others use data? 

I believe that superintendents must create a safe environment for data-driven decision-making. 
Superintendents must lead, model and encourage staffs to use different types of data and exam- 
ine the numbers systematically to avoid knee-jerk reactions to single, independent pieces of 
data. It will require training and trust so that every staff member can have access to, and look at, 
data on an ongoing basis. It requires a vision that is truly shared, so staff members are all contin- 
uously improving in the same direction. 



Not everything that is faced can be changed 
but nothing can be changed until it is faced . 

— James Baldwin, 
author 




One of the biggest challenges for any district implementing data-driven decision-making is 
knowing where to begin. Often, there are so many data to choose from that the process can be 
overwhelming for district staff, teachers and principals. The best advice is to start with the 
basics. Too often, school administrators and staff collect data without first figuring out what it is 
they need to know. 

"We recommend not fishing blindly as you review data," says Mike Parker, assistant director 
of the Center for Accountability Solutions at the American Association of School Administrators 
(AASA). "If you don't have a purpose in mind, it's easy to get off track." 

A district should begin data collection by defining what it wants to know. Is the strategic 
plan the district put in place three years ago improving student performance? Why are high 
school freshmen in your district garnering low grades in English? Did the district make good on 
its promise to the community that test scores would increase 5 percent over last year's scores? 
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In Data Analysis for Comprehensive Schoolwide Improvement, Bernhardt lists seven questions to 
help focus the early stages of data-driven decision-making: 

0 What is the purpose of the school or district? 

o What do you expect students to know and be able to do by the time they leave school? 

(Standards) 

o What do you expect students to know and be able to do by the end of each year? 
(Benchmarks) 

o How well will students be able to do what they want to do with the knowledge and 
skills they acquire by the time they leave school? (Performance) 

o Do you know why you are getting the results you get? 

o What would your school and educational processes look like if your school were 
achieving its purpose, goals and expectations for student learning? 

o How do you want to use the data you will gather? 



Chapter 2 takes a closer look at different types of data that can help answer these 
questions. Of course, the best questions are the ones superintendents, principals, teachers and 
school board members develop together. 

Making decisions based on data is a little like being a detective. Good data analysis requires 
asking lots of questions, uncovering more and more information and revisiting hypotheses 
along the way until a complete picture — supported by the facts — unfolds. 



Challenging Assumptions 

Data-driven school improvement requires 
administrators to challenge their own assump- 
tions. Almost every district has common 
beliefs about a school or groups of students. 
One school may be known for its nationally 
ranked reading program. Another may be dos- 
ing the achievement gap among different 
groups of students. But are these schools really 
accomplishing what they think they are? How 
do they know for sure? 

Data help district leaders determine 
whether their perceptions match reality. 
Running a longitudinal analysis, for example, 
will show whether a reading program is 



You use data to inform the doctor of the progress of the 
patient. You determine whether the patient is progressing 
in a good direction — or is there additional assistance that 
the patient needs? That analogy works for me, because 
that's really what you're asking for: You want data that are 
more diagnostic, that permit you to monitor progress on a 
regular basis and that provide you with the student's vital 
signs of learning. 

— fames Parsley, superintendent, 

Vancouver (Wash.) School District 
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Lessons Learned 
from Seattle 

Spokane (Wash.) 
Superintendent Brian 
Benzel learned the follow- 
ing lessons using data to 
drive decision-making in 
the Seattle (Wash.) 

Public Schools: 

• Start small; don't 
overwhelm staff with 
a "data dump." 

• Begin with the 
core issues, such as 
student achievement 
in reading or 
mathe-matics. 

• Listen to what the 
data tell about the 
big picture; don't 
get lost in too 
many details. 

• Work to create trust 
and build support by 
laying data on the 
table without fear of 
recrimination by staff. 

• Provide training 
opportunities for staff 
on how to use data. 

• Be patient, working 
with what is possible 
in the district.. 

O 



sustaining its impact over time. Disaggregating data by different student popula- 
tions will show which students are excelling and which are falling behind. These 
clues begin to form a picture of what is really happening in schools. 

In the rural community of Antigo, Wis., 90 miles northwest of Green Bay, data 
helped Superintendent Lance Alwin bridge the divide between educators and fami- 
lies that homeschool their children. The work began when the Antigo Unified 
School District reviewed progress toward its goals, one of which was to become "an 
inclusive educational community." 

"One of the questions that I asked was why we weren't providing services to 
families of homeschoolers," says Alwin. "The response was that homeschoolers are 
on the religious fringes. They want to do their own thing and they don't like what 
we have to offer." 

It was a stand off, Alwin recalls. But if the district truly was to be an inclusive 
community, it would have to tackle this issue head-on. 

Data about the district's homeschool population was collected and analyzed. The 
district surveyed its homeschool population, asking parents why they home- 
schooled their children. 

What the district learned challenged long-held perceptions. First, data revealed 
the number of families homeschooling their children was much higher than origi- 
nally thought. The data also showed that religion was not the number one reason 
parents homeschooled their children. Instead, parents said they felt they could do a 
better job than the public schools. District leaders also were surprised to learn that 
80 percent of homeschool families were interested in accessing instructional 
resources provided by the district, as long as no strings were attached. 

In response, district leaders created a unique charter school designed to meet the 
needs of homeschool families based on the information and data they had gath- 
ered. The school opened during the 1997-98 school year with four students. In 
2000-01, 62 children enrolled in the school. "We've had to turn students away," 
says Alwin. "Now, we're getting calls from homeschool families several counties 
away because they have heard that this school district is willing to work with 
them." 

The district is meeting its goal of becoming more inclusive. During the 1997-98 
school year, 178 children who were eligible to attend the schools in the Antigo 
Unified School District were homeschooled. During the 2000-01 school year, the 
number dropped to 119, bucking national trends. Alwin says confronting the data 
forced the district to examine new relationships. "This was rich data for us to 
mine," says Alwin. "It helped us determine how to create a charter school that was 
of use to them, not us." 
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Asking the Right Questions 

The more administrators dig for answers, the more questions emerge. As they analyze the data, 
they begin to see patterns and trends. They may notice a cluster of schools doing better than the 
rest of the district. Still more questions surface. What are these schools doing differently? Can 
these best practices apply to other low-performing schools in the district? Why or why not? 

Those who have experience using data say some questions will elicit more substantive infor- 
mation than others. In Using Data for School Improvement, a 1998 report by the Annenberg 
Institute for School Reform, Kate Jamentz, WestEd's director of programs in professional and 
organizational learning, says: 

For example, questions that ask "which" are better than those that ask "how 
many?" Asking which students are not meeting the standards in reading is bet- 
ter than asking how many students are meeting the standards. It is important 
to look at the types of questions we're asking because the nature of a question 
determines next steps in data collection and analysis. 

Phil Streifer, associate professor of educational leadership at the University of Connecticut 
and a data specialist, calls an in-depth line of questioning the "drill down" process. (See Chapter 
2 for more on the "drill down" process.) Streifer says the drill down process "starts with a 
global question or issue, which is then broken down into its component parts for analysis." 
Streifer says once the data analysis is completed, the team can make "a reasoned decision on 
next steps." 

Newark (N.Y.) Central School District Superintendent Robert Christmann agrees. "Data are 
only as good as the questions you are asking. You need your questions to be as specific as possi- 
ble so you can know whether your efforts are successful." 

Key considerations when formulating questions include: 

0 How should student achievement be measured in the district? 

0 Are goals for student achievement based on data elements aligned with what the 

teachers teach? 

0 What are the best indicators of student achievement upon which the district should 
base its decisions? 

0 What indicators of student achievement are collected regularly throughout the year so 
that informed decision-making can occur? 



Computer Software and Data-Driven Decision-Making 



Companies are creating software tools for superintendents and staff who are engaging in data- 
driven decision-making. AASA's Center for Accountability Solutions advises consumers to look 
for software that offers flexibility to customize district data reports. These reports should allow 
district leaders to review data for different student groups over time. 
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"Through our experience; we have found that each district's unique needs will determine 
which tool is best/' says Geannie Wells; the center's director. "There is no one-size-fits-all 
solution." 

Be sure to compare different software products. Talk with district staff, teachers and others 
about what capability the software should have to meet their needs. 

Districts should ask two sets of questions when purchasing data software — one of themselves 
and one of the vendor. 

District leaders should ask themselves: 

o What types of data should be included? 
o Who will need access to which data elements? 
o Is a product accessible through a web browser attractive? 
o Is the infrastructure available to house a web-based product? 

o Is it feasible to use an application service provider (ASP)? 

o is it acceptable to house the data outside the district on nonschool-system servers? 

o How many records and fields are needed? 
o How often will the district need to refresh data? 

o Should the software allow the district to create groups, select assessments and generate 

its own reports or should the district select software with existing templates? 

After deciding what the district's needs are, administrators can shop for software armed with 
a list of consumer-sawy questions for the software vendor, such as: 

o How will district data be transferred into the new software system? 

0 How many data updates are included in the software cost? 

o Who is responsible for importing new data? 
o What security features are built into the software? 
o What are the limitations on fields and records? 

o What type of, and how much, technical support will be provided by the vendor? 
o How much on-site training does the vendor provide? 

o When new software features are developed, are they provided free of charge or will 
districts have to pay for an updated version? 
o What type of database is required? 
o In what format are the data stored? 
o Can districts receive all of their data upon request? 
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What's Available? 



Increasingly, manufacturers of traditional student information systems are adding decision sup- 
port services. It is wise to research what features the district's software vendor might include in 
the future before purchasing new software or completely overhauling existing technology. 

Lots of data-specific software options also are on the market. Some packages offer a suite of 
software tools designed to help districts collect and analyze disaggregated data. These tools can 
collect and report longitudinal data on individual students, including demographic and testing 
information, or track different areas, such as safety and security, curriculum and instruction 
or parent involvement. 



Other web-enabled software allows districts to 
collect and analyze data that help them report on 
student performance and identify solutions. Still other 
software offers an analysis of questions tied to equity, 
as well as comparisons that highlight the effectiveness 
of programs, grade-level groupings, technologies 
in the classroom and more. 



Time is money in a school district. Using a computer ; 
software program that helps collect and analyze data 
allows us to spend that time and money more wisely, i 

— Roland Smit, superintendent , ; 
Mobridge (S.D.) School District [ 



Software prices vary widely. Some companies 

charge a fee based on enrollment, while others charge per school or site. Some districts require 
more customization than others, which drives the price up. Some software packages, such as 
Quality School Portfolio (QSP), are free at this time. 



A listing of many such products is available from AASA's Center for Accountability Solutions 
at www.aasa.org/data. 



Homegrown Efforts 

While some districts prefer to use commercial software, other districts are developing home- 
grown models using existing software tools to store and access data. Like their commercial coun- 
terparts, these tools disaggregate data and conduct statistical analyses. Typically, they can store 
multiple indicators of student achievement, everything from grades to dropout rates to annual 
assessments and more. 

As chief operating officer for the Seattle (Wash.) Public Schools before becoming superintendent 
in Spokane, Wash., Brian Benzel and his staff worked to create a "data decision support system." 

"First, we worked to create a profile of a Seattle Public Schools graduate/' Benzel says. "The pro- 
file reflects a series of statements about what we wanted our graduates to accomplish, including 
the importance of communicating clearly, thinking critically and working productively in teams." 

The district also examined its grade-level standards in mathematics, reading and writing to 
make sure students were learning the skills and knowledge they needed to reflect the Seattle 
Public Schools Graduate Profile. 



Using Data to Improve Schools: What's Working 



Benzel and his staff used multiple indicators to gauge whether teachers were using the most 
effective instructional practices and what impact those practices were having on student achieve- 
ment. The indicators are grouped into five categories, including academic achievement and stu- 
dent behavior. Test scores, attendance rates, truancy rates, student climate surveys, staff surveys 
focusing on areas such as teamwork, leadership and curriculum, and other data are used to 
measure overall progress. 

Benzel began developing a "value-added school indicators profile" that uses longitudinal data 
to predict student achievement. "We wanted to make sure that a year's worth of instruction was 

resulting in a year's worth of gains," Benzel says. 

The stakes are high in Seattle, where parents decide 
which school their child will attend. Funding follows the 
student. "In a choice system like [Seattle's], customers 
need to know how schools are doing so they can make 
wise choices," Benzel says. 

In Poway, Calif., just north of San Diego, the school 
district has taken another "homegrown" approach to 
data-driven decision-making. This suburban, upper-income district of 32,000 students began its 
foray into data-driven decision-making in 1996. The district had completed an intensive strate- 
gic planning process, but was struggling with how to measure whether it was achieving its 
vision, mission and core values. 

Director of Learning Support Services Ray Wilson says data analysis began with identifying 
what the district wanted to measure. Charting student progress was at the top of the list. "For 
example, there's research that shows that students who aren't reading at grade level by third 
grade will have problems later in school," Wilson says. "So, one benchmark we set was that all 
third graders would be reading at grade level before going to fourth grade." 

Decisions also were made about who would collect what data. The district agreed to collect 
benchmark data (e.g., third grade reading scores, fourth grade writing scores, fifth grade mathe- 
matics assessments, etc.) and provided school leaders with training and tools to help them col- 
lect and analyze their own annual data. A part-time data analyst was assigned to every school. 

"A principal does not have time in her day to sit and analyze data," Wilson says. "You can look 
at the data and see a problem, but investigating the 'whys' behind it can take hours and hours." 

Data-driven decision-making has prompted Wilson and his colleagues to redefine what quali- 
fies as data. Early on, they largely thought of it as student performance data — state and local 
test scores. It wasn't long before they realized they needed to broaden their definition. 

"For example, in a district like ours with a large white population, it's easy for the majority 
student population to mask the problems of minority students," Wilson says. "Our students 
score at the 80th percentile on norm-referenced tests, but when you disaggregate the data by 
race, gender, ethnicity and the number of years a student has been enrolled in the district, you 



We wanted to make sure that a year's worth of 
instruction was resulting in a year's worth of gains. 

— Brian Benzel , superintendent , 
Spokane (Wash.) Public Schools 



Chapter 1: Why Data Matter 



find small pockets of students performing at the 30th and 40th percentiles." In other words, it's 
not just about tracking how well students are performing on tests, but determining which 
students are not doing well and why, and then seeking improvements. 

Using data wisely requires clarity of purpose, time and a desire to seek and understand 
improved educational achievement. Ultimately, superintendents say that data have made their 
job easier because data help them determine whether the policies approved at the district level 
are having the intended impact in the classroom. 




Key points in Chapter 1 include: 

o Data provide quantifiable proof, taking the emotion and rancor out of the decision- 
making process. 

0 Determining what data to collect is based largely on first figuring out what is important 
to know about student performance, teacher quality, parent and community satisfac- 
tion and other issues tied to district goals. 

° Data-driven school improvement helps superintendents and others know whether the 
district and its schools are realizing their vision and purpose. 
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Using Date to Make 
Smart Decisions 



0 What different types of data should superintendents use when assessing 
student performance? 

0 How can data analysis effectively target student achievement gaps? 

0 Which methods work best for efficient — and accurate — data collection? 

0 What analytical methods can school administrators employ so they are 
confident that they are interpreting data correctly? 

0 How can superintendents develop an effective accountability program that 
is supported by staff, parents and the community? 



O ften, the term data is confused with the term statistics, but the two are not interchange- 
able. Data go beyond numbers, averages and percentages. They are the raw materials for 
effective decision-making. But to make the right decisions, superintendents need the right tools. 



No single set of data or collection method applies to all school districts. While districts might 
share common goals, demographics and achievement 
levels, they are not identical. Therefore, it is essential 
that districts consider their unique traits when devel 
oping their data systems. Such consideration will 
invariably help their efforts to improve student 
achievement. 

There are some commonalities in all data collec- 
tion. Successful integration of data-driven decision- 
making into educational strategy requires a team 

approach. That teamwork starts with the relationship between the board of education and the 
superintendent. The district's expectations must be clearly articulated, measurable — and attain- 
able. The roles of the school board and school staff must be clearly defined. And the superin- 
tendent — who ultimately is responsible for coordinating the data collection, regulation and 
reporting — must work collaboratively with the board, staff and community to lead the district 
toward improved student performance. 



| Collecting data without purpose is meaningless. j 

— Theodore B. Creighton , author , ; 

Schools and Data: The Educator's Guide for Using Data to \ 
! Improve Decision Making 

i ! 
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Data 




Before student performance can be improved; it must be defined and measured. Indicators of 
student performance include: 

0 Test scores 

o Rigor of coursework 

0 Graduation rates 

0 Attendance rates 

0 Promotion rates 

0 Rates of participation in co-curricular activities (including community service) 



By asking more questions — What percentage of ninth graders took advanced algebra? What is 
the program's gender breakdown? Are any of the students in an English as a Second Language (ESI) 
program? Is this the first honors course for any of the students? — superintendents get more com- 
plete answers. Time; staff capacity and cost factors prohibit unlimited disaggregation; but asking a 
few, specific; thoughtful questions provides better data. And better data lead to better decisions. 




Data collection starts with this question: What in particular does the district want to learn? 

Once this question has been answered; Phil Streifer ; associate professor of educational leadership 
at the University of Connecticut; recommends conducting a data audit to determine if the need- 
ed data are available. When beginning to search for the answers; superintendents should first 
look at their own information. Districts already keep track of certain data in different ways and 
in a variety of places such as teachers' files on students; individual school offices; district offices; 
even board member files. 

Existing data sometimes are found in digital format on spreadsheets. This is often the easiest 
and most efficient way to maintain data. While many spreadsheet applications are available, 
superintendents should consider using a program that incorporates or easily converts this data 
to graphics. The ability to present data graphically is essential — especially when dealing with 
community stakeholders and the media. (See sidebar on p. 34.) 



| Learning is not attained by chance, it must 
| be sought for with ardor and attended to 





Data can be measured on many levels. By looking at aggre- 
gate data, superintendents can form a general assessment of a 
particular curriculum, class or school. But breaking down, or 
disaggregating, data provides a more complete view. It goes 
beyond the "what" and tries to discover the "why" and "how." 
For example, 32 percent of ninth graders at Elmwood High 
School passed advanced algebra. Aggregated, the data do not 
bode well for the advanced algebra program. When the data are 
disaggregated, a more complete picture comes into focus. 
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In many cases, if the data are not immediately at hand, superintendents 
do not have to look far to find answers. "A lot of valuable data can be found 
if districts look at their own practices," says Theodore B. Creighton, associ- 
ate professor of educational leadership at Sam Houston State University. 
"There are a lot of data that we should collect that we don't collect." For 
example, tracking parental involvement — hours spent at school, Parent- 
Teacher Association participation, and so on — can easily be accomplished. 

That leads to another question: Are the data important for school 
improvement? Perhaps the parental involvement data are essential for one 
school, but the issue is not important for another. Even in the same district, 
schools have different data needs — and different ways of collecting data. 



Collecting New Data 

Although some data will already be available, not everything is easily acces- 
sible, and Streifer advises superintendents to consider the cost benefits of 
collecting new data before proceeding. "Many districts often want to add 
student tests to match state mastery testing in the 'off' years for longitudi- 
nal analyses. A decision should be made concerning the cost benefit of 
adding these tests in terms of lost instructional time and the cost of pur- 
chasing, administering, scoring and analyzing these tests." 

How do superintendents know what new data are worth generating? 
"This depends on the mission-critical nature of the question under consid- 
eration," says Streifer. "If the data needed are the only way to address the 
problem, and the question is mission-critical, then, yes, they would be 
worth generating." 

There are other considerations beyond cost: 

0 Are the data being used to achieve a district goal? 

0 Are the data the only way to address the problem? 

0 Given that it generally takes six months to three years to generate, 
collect and analyze data, can the district wait? Will the data still be 
relevant to the question? 



0 



Examples of Targeted 
New Data Sources 

Every district collects and ana- 
lyzes data on student test scores 
and grades. But there are other 
ways to measure student per- 
formance, such as: 

• surveys and questionnaires 
(of teachers, students, par- 
ents, employers, community 
members, etc.); 

• interviews or focus groups 
(with the same groups 
mentioned above); 

• teacher logs/diaries; 

• classroom observations of 
actual instructional practices 
and student responses; 

• alternative assessments (e.g., 
work samples, portfolios, 
senior projects and perform- 
ance tasks); and 

• locally developed pretests 
and posttests. 

Qualitative information, which 
describes what people actually 
say, do, think or feel, puts a 
human face on quantitative data. 
It provides depth and detail and 
may increase a person's under- 
standing of a situation. 

Source: At Your Fingertips: Using 
Everyday Data to Improve Schools, 
1998 
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Testing Results Are 
More Than Grades 

The most widely used — and publicized — 
method of assessing student performance is 
testing. But testing is not a one-size-fits-all 
endeavor. The different kinds of tests — 
each with their own guidelines — can be 
subject to different interpretation. 

0 Norm-referenced tests, the most com- 
mon form of educational achieve- 
ment tests used, compare individual 
student performance with a national 
sample of others in the same grade. 
Results often are described as "stu- 
dents scored in the 62nd percentile," 
meaning that they scored higher than 
61 percent of the students who took 
the test (Iowa Tests of Basic Skills, 
Stanford Achievement Test, etc.). 

0 Criterion-referenced tests measure an 
individual's performance against a 
well-specified set of standards. Results 
often are described as "62 percent of 
students met or exceeded the stan- 
dards" (National Assessment of 
Educational Progress, state proficiency 
tests, Advanced Placement exams, 
etc.). 

0 Performance tests require students to 
demonstrate their abilities (via portfo- 
lios, presentations, experiments, etc.). 

0 International tests, though not tech- 
nically a test "type," have gained 
prominence by comparing student 
performance by country (Third 
International Mathematics and 
Science Study, etc.). 



O 




Data Collection to 
Support Student Achievement 



Plainview, Okla. 

Data collection is not a one- 
size-fits-all process. And it is 
not limited to those districts 
where student performance is 
below standard. High- 
performing schools are 
finding that data collection 
and analysis can help them 
build upon an already solid 
foundation. 

Plainview School in south- 
west Okla. is a rural PreK-12 
school with approximately 
1,200 students. The school 
began extensive data collec- 
tion in 2000. 

"The most difficult part 
was finding what we wanted 
to analyze — and working 
with others on resistance to 
change," says Lisa Moore, 
Plainview's principal. "The 
easiest part was making the 
decision to look at our educa- 
tion process in order to 
improve." 

Plainview's accomplish- 
ments are impressive. In a 
ranking of Oklahoma's 547 
public schools, the school 
came in 11th in efficiency and 
18th in student performance. 
Several nationally certified 
teachers are on staff and, in 
1999, Plainview was recog- 
nized by Expansion Management 



magazine as a "Blue Ribbon 
Scholar School." 

Moore thinks that, in 
some ways, rural school dis- 
tricts have data collection 
advantages over larger dis- 
tricts. "Rural schools typically 
are smaller, thereby making it 
easier to obtain clear, close- 
to-the-student data," she 
says. "We have unique oppor- 
tunities — like close commu- 
nications — to facilitate data 
collection and usage." 

Plainview uses observa- 
tions, surveys and standard- 
ized test score analysis to 
collect data. The data are 
presented through reports 
on individuals, classes, grade 
levels and school buildings. 

Moore emphasizes that 
the purpose of the data col- 
lection and analysis is to 
enable the school to make 
appropriate decisions about 
curriculum structure and 
alignment. "The school 
board, with the help of the 
administration and faculty, 
uses results to ensure that we 
are meeting the needs of our 
students and steadily increas- 
ing our school improvement 
process to the maximum 
potential," she says. 
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It is important that tests be reliable and valid. Reliable tests are consistent; when different 
individuals review their results over time they arrive at the same conclusion. Defining validity in 
a test is not as precise. In essence, a test is valid if it measures what it claims to measure. 



Here's another way to think about it. Let's say a man steps on his bathroom scale every 
morning, and the scale reads 240 pounds. Each morning the result is the same, meaning the 
scale is reliable. When the man heads to the doctor for his annual check-up, he is weighed on 
the doctor's more accurate scale, 
and the results are different. This 
scale shows that the man actual- 
ly weighs 245 pounds, 5 pounds 
more than on his scale at home. 

Although the man's scale at 
home is reliable, it is not valid 
because it consistently underesti- 
mates his weight. All measures 
are subject to the same types of 
errors. Verifying both reliability 
and validity is critical to ensuring 
that tests actually are measuring 
what they purport to measure. 

When data show gaps in stu- 
dent performance, they show 
more than where a school or dis- 
trict did not meet its goal. When 
analyzed, these data show which 
student groups need more 
improvement, thereby enabling 
districts to address students' 
needs. For example, Elmwood 
High School's goal was for ninth 
graders to score in the 60th per- 
centile on the state mathematics 
test. That school year, the stu- 
dents scored in the 50th per- 
centile. While the school did not 
meet its goal, the results were 
not considered a failure, and the 
district set about to improve 
overall performance. Further 



The Annenberg Institute for School Reform has developed the Inquiry 
Cycle, six activities that attempt to capture an ongoing, nonlinear 
process that involves the interaction of reflection and action. Each 
activity is essential; none may be omitted. The inquiry process is not 
over after completing one cycle. The six activities are: 




The Inquiry Cycle 



Source: Annenberg Institute for School Reform 
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analysis showed that female students scored in the 40th percentile, while 
male students scored in the 60th. The question for the district changed: 
Why is there a difference in performance between male and female 
students, and how can this gap be closed? 



Indicators of student achievement also go beyond academic attainment. 
They include "softer" measurements. Qualitative measurements, 
in a sense, humanize the data by going beyond the numbers. These "quality- 
of-life" indicators include: 



Even rich assessment tools are relatively 
impotent if not embedded in systems 
committed to the standards they address , 
hungry for the data they produce , and 
willing to challenge and redesign traditional 
practices and organizational structures that 
get in the way of each student [meeting] 
those standards . 

— Kate Jamentz, director of programs 
in professional and organizational 
learning , WestEd 



job skills and preparation; 
citizenship (volunteerism, voting, etc.); 
appreciation of the arts; 

development of character and values (integrity, patriot- 
ism, work ethic, etc.; 

healthy lifestyle. 



and 



Education reform is — and will continue to be — scrutinized by 
the media, politicians and community stakeholders. Therefore, 
it is vital that any data collected be interpreted accurately and 
fairly. "Achievement gains tend to occur when a school ana- 
lyzes student performance data and develops a plan of action based on that 
analysis," says Daniel Domenech, superintendent of Fairfax County (Va.) 
Public Schools. "Conversely, schools that select programs based on 'per- 
ceived' needs almost always meet 
failure." 



The Vermont Department of Education recommends the following 
when analyzing student performance data: 

o Whenever possible, use data from the same sources from year 
to year. 

0 Collect and analyze as much data as possible. Use multiple measures. 

0 Always consider the nature, size and characteristics of the group 
being tested. 

0 Be sure to know what is being measured. 

0 Remember that large-scale assessment data are about the school's 
program — not just about the grade at which the assessment is 
administered. 



Chapter 2: Using Data to Make Smart Decisions 



o Compare performance from different groups of students (disaggregated data). The 

goal is to reduce differences in performance among groups while increasing excellence 
for all. 

Reacting to a single test score is perhaps the most common 
mistake made when interpreting data. Longitudinal measure- 
ment — conducted consistently from year to year — is neces- 
sary to properly measure progress, growth and change. The con- 
sistency of the data elements — necessary for any long-term 
data collection and analysis — reduces confusion. 

The level of analysis also can diffuse misinterpretation. The 
"drill down" process is an effective method of disaggregating data. It begins with a general 
question, then "drills down" the question into smaller and smaller parts. In Data Analysis 
for Comprehensive Schoolwide Improvement , Victoria Bernhardt describes four layers of 
disaggregation: 

First-Layer Disaggregations 

How many students are there? 

Male vs. female 

Limited English Proficiency (LEP) vs. non-LEP 

Ethnicities 

Lunch codes 

Second-Layer Disaggregations 

How have the demographics changed over time? 

Increases vs. decreases in categorical variables 

Third-Layer Disaggregations 

What percentage of students are gifted, and are they equally distributed among genders and 
ethnicities? 

Fourth-Layer Disaggregations 

How has the enrollment of LEP students entering the building changed over the years? 

Do students with higher attendance get better grades? 



If your data set is only test scores, you are 
doomed. 

— Suzanne Bailey, restructuring issues consultant, 
Tools for Schools, August/September 2000 
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"Walking Through" Data Collection 

Palisades School District, Pa. 



The Palisades School District is a 100-square-mile 
area in Upper Bucks County, Pa., with a population 
of approximately 15,300. It serves 2,200 students in 
three elementary schools, a middle school and a 
high school. 

When Francis Barnes took the job as superin- 
tendent, he was charged with finding out why stu- 
dents received good grades on report cards but 
performed poorly on standardized tests. Then 
Barnes had to develop a plan to improve student 
performance. 

The first task at hand was to collect existing 
data. "There were a variety of pieces of data in a 
number of places," Barnes says. "The problem was 
that the data were not organized sufficiently to 
permit any analysis. Nor did the administrators or 
staff have training in the analysis of data for the 
purpose of improving student performance." 

Barnes and his staff developed an action plan 
that was aggressive, yet thoughtful and realistic. 
"We generated, by consensus agreement, a data- 
driven decision-making model," he says. "Our data- 
driven process engages stakeholders in decision- 
making to attain district goals. This data-based 
decision-making vision required us to gather all the 
available data and analyze the information, seeing 
what we actually had and also what was missing." 
This included mainly state test and standardized 
national test data. 

The district then participated in a pilot project 
with the AASA Center for Accountability Solutions 
to use Quality School Portfolio (QSP). This project 
provided software for staff to input and organize 
existing data, and to develop other assessment 
measures to fill existing gaps. 

"We believe that both qualitative and quantita- 
tive data are essential," Barnes says. With this in 



mind, the district applied a hospital model for 
assessing its educational health. "We used quantita- 
tive measures, such as test score results, as indica- 
tors of academic health in the same way physicians 
use numbers, such as temperatures and blood pres- 
sure readings, as measures of physical health. We 
also used softer qualitative measures, such as the 
result of interviews with students, in the same way 
physicians ask questions during physical examina- 
tions." 

Some of these softer qualitative measures are 
unique to Palisades. "Walk Through" data, a term 
coined by the district, are gathered by interviewing 
each student once each year using a faculty-devel- 
oped protocol. (See the list of questions boxed on 
p. 21 .) "The interview results give us qualitative 
data as to whether students know what we expect 
of them in writing, reading analysis and math; 
whether they know how their work is scored; and 
whether they know how to improve it," Barnes 
says. Responses are analyzed at the building level, 
thereby enabling faculty to assess whether students 
are learning what is expected. The data then are 
used to make necessary changes in curriculum and 
instruction. 

As Palisades' data-driven decision-making 
efforts continue, its data-collection methods have 
expanded. Barnes notes districtwide assessments in 
kindergarten through eighth grade are the direct 
result of data analysis. The district currently is for- 
malizing assessments in 9th through 12th grade. 

The effort is producing results; Palisades School 
District students are scoring higher on standardized 
tests. For example, in 1998, Palisades High School 
students scored an average of 1 270 on the 
Pennsylvania System of School Assessment (PSSA) 
reading test and 1250 on the mathematics test. 
(The state's "proficient" score is 1 300.) By 2001, 
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the students 7 average scores increased to 1 380 on 
both tests. 

The district does not rely on just one data 
resource, however. "We use the state assessments in 
reading and math in grades 5, 8 and 11, and writ- 
ing in grades 6 and 9," says Barnes. "While these 
are 7 good 7 assessments... we were somewhat frus- 
trated in the 1996-97 school year that, while these 
tests measured student performance, we did not 
receive any data that would help us improve 
performance." 

The district subsequently adopted the New 
Standards Reference exams, which were adminis- 
tered for the first time in spring 1998 to students in 
grades 4, 8 and 10 in reading and mathematics. 
"These tests provide specific information by student, 
class and district for areas that need to be 
improved," says Barnes. "We have taken this data 
seriously and provided the appropriate training to 
prepare our teachers to use the data to improve 
instruction." 



Barnes says the data indicated that students 
needed to work on writing conventions, reading 
analysis and interpretation, and math problem solv- 
ing. So he focused the first year on writing, the sec- 
ond year on math problem solving and the third 
year on reading analysis and interpretation. "Focus 
means that each year we provide training and time 
for administrators and faculty to work on imple- 
menting changes in their instruction based on what 
they have learned in the focus area," says Barnes. 

Faculty also have created quarterly assessments 
in math and language arts at each grade level to 
provide benchmark data to use for improvement. 
When students do not achieve on quarterly assess- 
ments, they participate in after-school and even 
summer-school tutoring sessions. The results are 
impressive. For example, in spring 1998, 33 percent 
of the high school students taking the New Stand- 
ards exams for reading analysis and interpretation 
scored at or above the standards. In 2001, that fig- 
ure was 64 percent. For mathematics problem solv- 
ing, the number of high school 
students scoring at or above the 
standard increased from 18 per- 
cent to 43 percent. 

Results also are evident in stu- 
dents' accomplishments after they 
graduate high school. They are 
entering better postsecondary 
institutions; students from 
Palisades High School's Class of 
2001 now attend such prestigious 
schools as Boston University, The 
College of William and Mary, 
Franklin and Marshall College, 
Temple University and Le Cordon 
Bleu. And Barnes notes that the 
community is showing more 
interest in and support of stu- 
dents, staff and the district's 
efforts to help all students fulfill 
their potential. 



Elementary Reading Analysis and 
Interpretation Walk Through Questions 

Durham Nockamixon Elementary School, Palisades School District, Pa. 

• Can you share a connection you have made with text 
this year? 

• How did that connection help you understand the story? 

• What clues do you use to make predictions when you 
read a story? 

• What kinds of questions do you ask yourself before, 
during and after reading a book or story? Can you share a 
question you've written down? 

• What do you do when you do not understand what you 
are reading? How would you help a friend who didn't 
understand what he was reading? 
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Accountability Plans 

School districts have been mandated to improve student performance. Domenech, whose dis- 
trict is responsible for more than 158,000 students, shares his experience: "To blunt the criticism 
of a community that implied the schools constantly asked for more money but were never will- 
ing to be held accountable, we devised a plan that allowed for objective measurement of our 
performance and we agreed to be held accountable for our results, or lack of." 

Fairfax County's plan includes a process for determining a biennial objective and measurable 
targets that are part of a system that provides each school with specific and measurable targets 
for student achievement. Rewards and sanctions apply to schools and students. For schools, the 
rewards include bonuses; the sanctions include reconstitution. For students, there are promotion 
and retention guidelines by subject and grade level. Students who fail to meet the benchmarks 
are required to attend remediation programs and summer school. 

Implementing a full-scale accountability plan is not easy. "It is an education process and a 
change in culture," says Domenech. "We had the immediate support of the school board and 
the community. Getting the support of the administrators and staff was more difficult. Everyone 
had an excuse as to why objective measurement of our performance was not fair." 

Armed with the right data and a full understanding of what the data show, superintendents 
can effect change as well as measure progress. Says Domenech, "The use of gain scores tended to 
neutralize the socioeconomic factors and did away with most objections. The reward system also 
helped, although the sanctions created a great deal of angst." 




It is important to get community ownership and buy-in at the front end of developing an 
accountability plan — a method by which districts, schools and students are evaluated and held 
responsible for performance. Stakeholders should have input into deciding what counts and 

what districts should measure. By including all stakeholders in 
these decisions from the start, school districts benefit from their 
support throughout the process. 

In 1995, the Plainfield (N.J.) Board of Education adopted six 
goals focusing on school reform. Those goals included setting 
standards, identifying and establishing benchmarks, improving 
evaluation of personnel and providing skills development, and 
completing a community planning process that would result in 
a comprehensive district improvement plan. 



All stakeholders take ownership and respon- 
sibility for our children's achievement. We 
are convinced that community engagement is 
necessary in order to progress. 

— Inez P. Durham, superintendent, 
Plainfield (N.J.) Public Schools 




The district spent the next two to three years laying the 
groundwork for standards-based reforms, according to Joshua P. Starr, Plainfield's director of 
accountability. Most of the efforts were aimed at building community engagement. District lead- 
ers explained to parents, business leaders, and other community members what the reforms 
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were and how the reforms would improve student achievement. 

They also listened closely to community concerns. 

The district's efforts were effective: In 1998, voters approved 
on the first ballot a $33.9 million education referendum that pro- 
vided the necessary resources for expanding the district's educa- 
tion reforms. That same year, the board adopted a specific goal to 
deal with reforms based on research. "This meant that resources 
had to be put into place to ensure the efficient, effective gather- 
ing and analysis of data," says Inez P. Durham, superintendent of 
the district, which serves more than 7,200 students. In 1999, the 
board established an office of accountability. "From 1995, it was 
clear that all stakeholders had a responsibility for student 
achievement and school reform — and would be accountable." 

Plainfield is required by a court case to have a districtwide 
plan. The plan's acceptance and success is contingent upon sup- 
port from all stakeholders. Therefore, members representing the 
community, administration, parents and other groups were given 
the opportunity to participate in the plan's development. The 
resulting plan, approved by the board, has been well-received. 

"The plan is designed to promote local ownership and collec- 
tive responsibility," Durham says. "It is locally driven and 
identifies the outcomes and indicators that the Plainfield 
community values and expects. The system is designed to sustain good practices, continual eval- 
uation and continuous improvement based on reliable and comprehensive data. It is a system of 
rewards and consequences." 

Plainfield's accountability plan is implemented throughout the district via its Leadership 
Innovation and Change Councils (LINCCs), which are site-based management teams that 
include representatives from administration, teaching staff, support staff, parents, community, 
business and corporations and, when appropriate, students. 

In addition to a districtwide LINCC, each school has its own LINCC. The school-level LINCCs 
measure progress toward achieving six non-negotiable goals. There are six scoring guidelines for 
each goal. "It has been a challenge for all to become comfortable with following the procedures 
for collecting and reporting accountability data and to score utilizing rubrics," Durham says. The 
district is helping LINCC members overcome those obstacles with dry runs, inservice programs 
and workshops. 



Establishing Performance 
Targets 

Four main sources provide helpful informa- 
tion when establishing performance targets. 

Superintendents can: 

• Review relevant state and national 
performance standards. 

• Discuss expectations with important 
stakeholder groups. 

• Compare students' performance 
with that of an exemplary school 
or program. 

• Compare students' performance with 
state and national averages. 

Source: At Your Fingertips: Using Everyday Data 

to Improve Schools , 1998 



The plan has widespread support and interest. "All stakeholders take ownership and responsi- 
bility for our children's achievement," Durham says. "We are convinced that community 
engagement is necessary in order to progress." (See Chapter 3 for more about community 
engagement.) 
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A Closer Look 

Phil Streifer 

Associate Professor of Educational Leadership , University of Connecticut 

What is the first thing superintendents should consider when they develop a data-collection program? 

Data collection should start with the questions they most want to address. Ask: What data do 
I need to answer these questions? This will quickly lead to a list of the data needed — 
something I call a data audit. 

An issue of concern always is "What will it cost to collect the data?" I encourage superin- 
tendents to consider the cost benefits of collecting data in terms of the value they will provide 

in answering their questions. In some cases, questions may 
need to be modified because the data will be too costly to col- 
lect. For example, surveying large groups of constituents can 
be time-consuming and costly. It may be appropriate to do so, 
but only after thoughtful review of whether it's absolutely 
necessary or whether some other sampling might be equally 
valuable. 

There are different types of data. How do you distinguish 
among them? 

Data can be separated into three broad categories for analysis 
and interpretation: input, progress and outcome variables. 
Input variables are variables that students come to school with, such as their background, pre- 
learning, socioeconomic status and so on. Input variables also can include teacher variables, 
such as their level of education or previous in-service training experiences. Process variables 
are those that currently are at play on the issue under consideration, such as the curriculum 
in use, instructional techniques and materials. The levels of teacher and parent participation 
in decision-making also are process variables. Outcome variables are the results or measures of 
both input and process variables, typically student tests, observation data and survey results. 

A number of school districts do not have a districtwide accountability plan. Where should they begin? 

I would start with the district goals and build an accountability plan around those issues. 
Second, I would make sure that there is a high degree of match between the data needed and 
the data collected. Generally, this issue of match is first measured by the data's reliability (abil- 
ity to consistently yield the same result over time) and validity (the degree to which the data 
measure what they purport to measure). Finally, match is determined by the level of appropri- 
ateness of the data to your population. 



I think it's also important to desensitize 
the annual reporting of high-profile data by 
having regular reports to the board , staff and 
parents on student performance covering a 
wide range of variables. 

— Phil Steifer, associate professor of 
educational leadership , University of Connecticut 
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What types of data are necessary to create an effective accountability plan? 

I would focus first on district/board goals or mission-critical problems. Next, I would generate 
data that allow for longitudinal analyses — over several cohorts (groups of individuals having 
a statistical factor, such as grade level, race or age) if at all possible. Thus, if mastery tests are 
administered in the fall, I would look to a comparative assessment in the spring, or one that 
would allow for fall-to-fall analyses. Overall, longitudinal analyses yield the most useful infor- 
mation. Beyond this, I would ensure the data you have or are about to collect will provide 
you with the information needed. Poorly constructed surveys, for example, yield no useful 
information at all. Tests that are too easy or too hard also have limited utility. 

Board , staff, parent and public support are essential. How can school administrators get community 
buy-in for such a plan? 

I don't think you can get support after the fact — that is, when the results are in. Early on, we 
need to engage these folks in why these data are needed and in helping them to understand the 
data's limitations and value. I think it's also important to desensitize the annual reporting of 
high-profile data by having regular reports to the board, staff and parents on student perform- 
ance covering a wide range of variables. With consistent, regular updates from the district, the 
once-a-year state report may not take on the degree of importance it sometimes does now. I 
have found all of these constituencies to be thoughtful and reasonable once we take the time to 
explain the complexities and how the data are used to gauge progress and design improvement. 




Key points in Chapter 2 include: 

0 Districts should use multiple assessment and nonassessment measures when assessing 
student performance. 

0 Disaggregated data is a powerful tool to help superintendents identify which students are 
achieving, which students need additional assistance and how best to target limited 
resources. 

0 Methods for data collection vary; many superintendents start with the question: What 
do I want to know and what data will help answer this question? 

0 A districtwide accountability plan provides for objective measurement of performance 
and holds boards, superintendents, principals, teachers, parents, students and others 
accountable for results. 



O 

ERIC 



32 



25 



Chapter 



Data, Pybflk Edik 
itirateoic 



mmt 



Communication: 



0 What data matter most to your community ? 

0 In what ways is the community involved in discussing data, such as test 
results, school safety reports and teacher quality ratings? 

0 How is community input on how to improve schools gathered? 

0 How do teachers, staff and board members communicate with parents in 
the district? 

0 How is jargon-free data reported to the community, both the successes and 
the challenges? 

0 Who are the most credible messengers in the district? 

0 How can the district build an effective working relationship with the media? 



S uperintendents understand the importance of parent and community engagement, strategic 
communications and working with the media. They know that, unless they communicate 
clearly and honestly about data and what the data reveal about student learning, their efforts 
will fail and their credibility could be damaged 
irreparably. 

"School improvement requires a constant conver- 
sation with our community," says Beaverton (Ore.) 

School District Superintendent Yvonne Katz. "The 
community's input informs our processes, procedures 
and what we do. I'm convinced I wouldn't be here if I 
didn't gather and use community input." 

Success in promoting and sustaining data-driven 
decision-making as a tool for school improvement depends on: 

0 Educating parents and community members about what different data convey and 
how to use data to measure the success of school-improvement efforts, budget deci- 
sions, teacher quality initiatives and other practices. 

0 Asking teachers, parents and community stakeholders what their concerns are for 
schools and what data they need to see to know schools are improving. 







: 

i To depend on the local newspaper editor or reporter 
; to do all of your public relations for you is like 
: depending on the hospital orderly to do your 
! open-heart surgery. 

— Brian Hale, member, Neb. School Boards Association 



33 



27 



Using Data to Improve Schools: What's Working 



o Listening to the insights of these different stakeholders and reporting back to them 
how their input may shape decision-making, 
o Sharing the district's successes and failures along the way. 
o Speaking in clear, easy-to-understand language that is free of "educationese." 

o Understanding that the public schools belong to the public, and school leaders need 
the public's permission to do the work they are charged with doing. 

This chapter explores three cornerstones of effective communication: public engagement, 
strategic communications and media relations. 



Public Engagement: Listening to Your Community 

How do superintendents proactively involve their communities in understanding and asking 
informed questions about data? How do they determine which indicators matter most to various 
stakeholders — parents, taxpayers, teachers, business leaders, senior citizens and others? How do 
district leaders build support early on so that when controversial data are released, the public 
says, "How can we help?" rather than assigning blame or finding fault? 

"What's really important is to clarify what the community's role is," says Superintendent Jane 
Hammond of Jefferson County (Colo.) Public Schools. "We gather input from the community 
and then we get together with educators to decide what we want to do to get the results the 

community wants." 

Engaging the public begins with listening. It's not 
uncommon for an educator's view of success to differ 
significantly from the views of a taxpayer, parent or 
business leader. (See Communicating Data to the 
Public on page 32.) Yet each of these constituents has 
a stake in public schools. 

One of the most effective ways superintendents 
and district staff can engage their public is by asking: 
What would you need to see that would indicate our schools are improving? Other questions 
include: How do we as a community define success? What elements are important in defining 
student success? How do we as a community work together to ensure success for our young peo- 
ple? The answers to these questions not only give superintendents a clearer sense of the public's 
priorities, but they also identify what data are most meaningful. 

"When asked what their priorities are for spending money, our local community said 'we 
want a quality teacher in every classroom, including giving teachers the coaching they need to 
improve,"' Hammond says. "Professional development emerged as a higher priority than reduc- 
ing class sizes. I have never before seen the public be willing to support staff development." 



The fundamental purpose of any public engagement 
initiative is to channel a community's concern , apa- 
thy or anger into informed and constructive action. 

— Annenberg Institute for School Reform, Reasons for Hope, 

Voices for Change, 1998 
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The most effective strategies for community listening are face-to-face meetings, such as: 

0 Focus groups, typically involving 8 to 12 people. Some focus groups are conducted with a 
cross-section of people representing the community or a specific subgroup, such as parents 
with children in public schools or recent high school graduates. A skilled, neutral modera- 
tor leads the discussion using a script and predetermined questions. The moderator listens 
for key themes, new ideas, differences of opinion and common ground. 

0 Community forums/town hail meetings, 

which may attract anywhere from 60 to 200 peo- 
ple. Typically, these meetings focus on one issue or 
pose a question such as, "What should students 
know and be able to do?" or "What does a good 
school look like?" The information gathered at the 
forum can be used to help inform the district's 
strategic-planning process, goal-setting or data- 
gathering efforts. 

o A study circle involving parents, educators, cler- 
gy, students and other publics. Together, they use 
dialogue to arrive at a decision about an issue or a problem. A study circle typically is made 
up of 10-15 people who meet weekly for two hours over four weeks. A facilitator leads the 
discussion, often drawing from a discussion guide shared with participants. During the final 
session, the group decides how to address the issue and offers solutions. 

0 High-level advisory committees or superintendent leadership councils made up 
of a cross-section of business people and community leaders who periodically meet with the 
district superintendent. These committees offer guidance on key issues, provide a heads-up 
about what they are hearing in the community and help brainstorm alternative methods of 
public engagement. The councils can be issue-specific, addressing an issue such as how to 
win support for an upcoming bond issue, or they can be more general and focus on broadly 
defined areas such as leadership. 



L 



If you don't listen to your community \ you may 
make some assumptions that you are meeting their 
needs, but you are not. I find public engagement 
makes our decision-making easier, otherwise 
you are just guessing what the public wants. 

— fane Hammond, superintendent, 
Jefferson County (Colo.) Public Schools ; 



As school leaders begin to implement these strategies, f 
the challenge becomes how well they connect their 
work to the priorities of the community. Lackluster pub- 
lic engagement efforts can result in better packaging of ’ 
more of the same rather than increasing student learn- 
ing by using good data, gathering community input and 
making strategic decisions based on both. 



I don't think you'll survive without engaging your 
public. The public has to be in public education. 

—Yvonne Katz, superintendent, 
Beaverton (Ore.) School District 
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Engaging the Public in Strategic Planning 

Rochester (N.Y.) City School District 



Since Superintendent Clifford ]aney arrived in 
Rochester, N.Y., in 1995, his work has been guided 
by a simple but powerful idea: If community 
members and educators have a shared under- 
standing of what schools should do, they can work 
together to make that happen. Realizing the 
promise of that idea has not been easy, but the 
payoff has been enormous for the district's nearly 
37,000 students. 

At the start, ]aney found a community demor- 
alized by a history of low achievement. Experiment 
after experiment had been tried, but no "silver 
bullet" of school reform had pulled the communi- 
ty together to improve schools. ]aney believed that 
Rochester was "idea-rich and focus-poor." 

To help build a common understanding of 
what the community wanted from its schools and 
how that vision could be made a reality, ]aney 
invited the community to help set specific, meas- 
urable goals for schools. He met with scores of 
Rochester residents and community leaders to 
learn their priorities for the district. 

Next, ]aney worked with the board of educa- 
tion and district staff to formulate a set of per- 
formance benchmarks — goals for the district to 
meet by 1999 — based on the public's concerns. 
Detailed in the plan were 40 specific indicators of 
success in the four major areas that community 
members had highlighted: 

• Student achievement and citizenship 

• Safe and supportive schools 

• Community-school partnerships 

• District accountability 



But the community's participation didn't stop 
there. As part of this plan, the district secured 
commitments from community partners, including 
city and county government, social service agen- 
cies, colleges and universities and the private 
sector. A coalition of local groups sponsored an 
annual event to chart progress, raise awareness 
about these efforts and share information on 
district performance. 

During the next couple of years, Rochester 
made progress toward nearly all of the bench- 
marks, and public sentiment toward the district 
improved. When janey decided it was time to 
revisit the district's strategic plan, he stuck with 
the strategy of community involvement that had 
worked in the past. 

To assess whether the original benchmarks 
were still relevant, the school conducted focus 
groups and interviews with more than 200 teach- 
ers, administrators, parents, civic leaders, employ- 
ers, religious leaders and local politicians in 1997. 
Community members voiced support for the 
benchmarking initiative, but they said the original 
benchmarks were not ambitious enough, not 
streamlined enough and not focused enough on 
core academic skills. 

Rochester education leaders spent much of 
the next few years processing this feedback, 
retooling the strategic plan and gauging the 
district's progress. The district also conducted 
public opinion surveys to evaluate the attitudes 
of parents, students and school staff about 
district efforts. 
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Based largely on community input, the district 
unveiled revised benchmarks in the 2000-01 
school year. District officials had cut the number of 
benchmarks and honed their focus. These revised 
benchmarks revolve around two goals: 

• All students will meet high standards in 
reading, writing, mathematics, science and 
social studies. 

• All students will graduate prepared for 
civic participation, higher education and 
the world of work. 

Buttressing these goals are two new, specific 
benchmarks that describe the progress the district 
is expected to make between the 1999-2000 and 
2004-05 school years: 

• Within a given group of students who 
enter high school together in ninth grade, 
the percentage of students receiving a 
diploma within four years will increase 
from 44 percent to 90 percent. 

• The percentage of students receiving a 
Regents diploma with honors (indicating 
those students passed the New York 
Regents exams with an average of 90 
percent or higher) within four years will 
increase from 3 percent to 1 5 percent 

Rather than erecting walls to shut out the 
community, janey and other Rochester school offi- 
cials repeatedly invited the public to join in the 
work of strengthening education. The results have 
been sweeping changes and steady improvements. 
The district has not yet met all its goals, but it has 
made progress toward every benchmark set by the 
community — and it remains committed to com- 
munity engagement as a way of guaranteeing that 
public schools serve the public. 

Excerpted from Danforth Foundation Report , 
Rochester City School District , N.Y, 2001. 
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Crafting a 

Communications Plan 

The following are vital elements of a 

communications plan: 

• Messages: What do we hope to achieve 
as a result of communicating? 

• Audiences: With whom do we want 
to communicate? 

• Results: What measurable effect do we 
hope to achieve? 

• Medium: What communications method 
do we want to employ? 

• Timeline: By what date do we want to 
be successful? 

• Expense: How much are we willing to 
spend on communications? 

• District and school staff: Who will 
be responsible for successful 
communications? 



Engaging Business Leaders 

Who knows a community better than the 
community itself? The Florida Chamber of 
Commerce is exploring new ways of listening 
to its community. One idea is to use volun- 
teers as "listening posts" who keep their ears 
close to the ground. Bus drivers, hairdressers 
and others in the community could make it 
their job to pick up rumors circulating about 
the school district. When needed, they could 
stop the rumor before it can spread by offer- 
ing the "real story" about student achieve- 
ment, teacher recruitment, a bond issue or 
whatever the hot-button topic might be. 
These "listening posts" could also be con- 
vened periodically to discuss issues as a 
group or report to each other as needed. 
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Who Has Credibility in 
Your District? 

Too often, superintendents count on 
the media to share district news and 
information with the public. While the 
media play an important role, surveys 
show teachers and other staff members 
have more clout with parents seeking 
information. 

A 1 996 survey conducted by the 
Denver-based Education Commission of 
the States shows teachers have the most 
credibility among parents. Here's how 
the survey ranked who parents rely on 
"heavily" or "somewhat heavily" as 
sources of information about education 
issues: 



88% Teachers 
83% Children 
82% Other parents 
72% School officials 
68% Print media 



Source: Listen, Discuss , and Act: Parents' and 
Teachers' Views on Education Reform, 1 996 



Communicating Data to the Public 

An effective way to build public support and increase 
community confidence in public schools is to show the pub- 
lic how schools are being held accountable for results. Sharing 
data in easy-to-read charts and short, jargon-free reports helps 
deepen community understanding of the issues facing public 
education. 

A common concern voiced by superintendents is that the 
community reads limited coverage in the local newspaper 
about its school district or that reporters only cover the "bad" 
news. But too often educators make the mistake of placing 
too much stock in media reports. Polls show parents most 
often get their information from teachers, the principal, other 
parents and their children. Districts play an important role in 
disseminating factual, timely information to parents about stu- 
dent achievement — the information parents care about most. 



If we are interested in community 
building , then we, along with members of 
the proposed community, are going to have 
to invent our own practice of community. 
Despite the difficulties, if we are success- 
ful, our community will not be counterfeit 
but real 

— Thomas Sergiovanni, 
Building Community in Schools , 1994 



Increasingly, districts are reporting information directly to 
their communities with the help of a school report card or 
annual performance report. Many successful districts use the 
report to guide discussions on performance and education 
priorities at school forums, parent meetings, staff meetings 
and board meetings. Training teachers and principals to help 
facilitate these conversations ensures everyone is focused on 
the data, what the data reveal about performance and how to 
improve instruction in the classroom. 
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"We have spent countless hours and thousands of dollars training administrators on how to use 
data/' Katz says. "We know who is doing well in what area and who is not doing so well. We talk 
about the data and don't push the numbers under the rug. You cannot be a smoke and mirrors 
superintendent anymore in this country. People are very smart about wanting to see the evidence." 

A critical first step of reporting data to the community is finding out which data different 
stakeholders care about most. Often there are discrepancies among constituencies. For example, 
educators may want to see more demographic data, while parents tend to rank school safety and 
achievement high on their list. 

A national research project published in Education Week in 1999 (see Appendix A) asked par- 
ents, taxpayers and teachers across the country what indicators they would use to hold their 
schools accountable. Here is a sample of how parents, taxpayers and educators ranked their top 
five indicators: 



Parents 

School safety 
Teacher qualifications 
Class size 
Graduation rates 
Dropout rates 



Taxpayers 

School safety 
Teacher qualifications 
Graduation rates 
Dropout rates 

Percentage of students 
promoted to the next grade 



Educators 

School safety 
Class size 

Teacher qualifications 
Graduation rates 
Per-pupil spending 



Websites as a Communications Tool 



More and more school districts are using the 
Internet as a tool to communicate district data to 
their communities. Achievement data for stu- 
dents in Orange County (Fla.) Public Schools, 
Palisades (Pa.) School District and Plano (Texas) 
Independent School District are only a click away 
for parents, business leaders, teachers and any- 
one else who has access to the World Wide Web. 

Orange County's website — 
www.ocps.kl 2.fl.us — boasts detailed school-by- 
school enrollment data, scores from state and 
voluntary tests, an online parents guide and 
detailed frameworks of higher achievement in all 
subjects at every age level. 



As comprehensive is Palisades School 
District's site — www.palisadessd.org — which 
posts results by grade and school for the stan- 
dards in reading, writing, mathematical skills, 
mathematical concepts and problem solving, as 
well as the percentages of graduates attending 
two- and four-year colleges. 

District data also is a main highlight of the 
Plano School District website — www.pisd.edu. 
The district's Department of Research, 
Assessment and Evaluation posts assessment 
results, the yearly district accountability report 
and detailed explanations of all tests adminis- 
tered in the county. 
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Graphical 

Representation 

"Graphics reveal data/ 7 notes 
Edward R. Tufte in The Visual 
Display of Information. According 
to Tufte, graphics communicate 
complex ideas with clarity, precision 
and efficiency. Graphical displays 
should: 

• show the data 

• induce the viewer to think 
about the substance rather than 
about methodology, graphic 
design, the technology of 
graphic production or some- 
thing else 

• avoid distorting what the data 
have to say 

• present many numbers in a 
small space 

• make large data sets coherent 

• encourage the eye to compare 
different pieces of data 

• reveal the data at several levels 
of detail, from a broad overview 
to the fine structure 

• serve a reasonably clear pur- 
pose: description, exploration, 
tabulation or decoration 

• be closely integrated with 
the statistical and verbal 
descriptions of a data set 



Strategic Communications: Getting the Most 
for Your Effort 

Districts have a wealth of data to share with their communities: dropout 
rates, test scores, attendance rates, school safety data, budget figures and 
other information. One of the biggest 

challenges facing districts is determining where to begin. What informa- 
tion is most useful to communicate? Superintendents do not have the 
time or the resources to communicate with everyone about all of the dis- 
trict's initiatives. That is why it is important to be strategic. Simply said: 
Set priorities, gather input early on, plan ahead, be proactive and always 
evaluate the results of communication efforts. 

Because so many different initiatives and programs compete for atten- 
tion, it can be challenging for superintendents to keep their communica- 
tion efforts focused on improving student performance. District leaders 
who have turned around poorly performing schools say they worked 
hard to keep their communication with staff, parents and the public 
focused on achievement. 

For example, if a district's emphasis is on helping all students meet 
standards, school visitors should be greeted with student work that 
meets the standard, scoring guidelines outlining academic expectations, 
and students who can explain what they need to know and be able to do 



User-Friendly Report Cards 

Here are some helpful guidelines to ensure that a school's 

report card is read and used. 

• Keep school report cards short, such as a six-panel 
brochure. Have a more detailed version available for peo- 
ple who want more data. 

• Add short narrative descriptions. School data are not self- 
explanatory to nonexperts. 

• Advise parents and others how they can use the data. It's 
not obvious to most. 

• Spend as much time working on distribution as on pro- 
duction. Research shows that even most teachers say 
they've never seen district report cards. 

Source: Reporting Results: What the Public Wants to Know About Schools, 

1999 
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at each grade level. Schools can illustrate the progress students are making on tests that measure 
whether they are meeting the standards with the help of easy-to-read bar graphs and charts that 
help explain what data the district is using to measure progress. This approach helps parents and 
community members understand how these different efforts are connected. It also shows them 
that they are receiving important and timely information on school quality. 



Taking the Pulse of the Community 

A poll is a structured method to determine the level of support or intensity of opinion on an 
issue, product or philosophy. Public opinion polling is one way to gauge the community's views 
on education issues. For example, educators can use a poll to quantify the level of support for a 
proposed ballot issue or to test different options for a new start time for school. Results of 
polling should be used to inform decision-making, not as the sole basis for decisions. 

When considering whether to use polling, superintendents should think about the resources 
required (it can be expensive), the importance of the information to the decision-making 
process and the specific information needed so they'll know more than they did before they 
tested the waters. 



Here are some key points to remember when 

writing a poll: 

0 The way a question is asked can determine 
the answer. 

0 It's important to look for obvious questions 
that are not asked by the pollster. 

0 Respondents are more likely to tell pollsters 
they support plans favoring change, 
even if they do not fully understand how 
the proposed change will affect them. 

0 Not all pollsters make sure respondents 
know enough about the subject of the poll 
questions to give meaningful answers. 

0 There are many diverse publics, as well as 
variety in thinking among specific sub- 
groups of a given population. 
Superintendents must research what each of 
these constituencies thinks, feels and wants 
before making major decisions. 





School Ambassadors 

The National School Public Relations Association 
says the following school employees, in order, 
are best known to the public: 

• School secretaries 

• Custodians 

• Bus drivers 

• School nurses 

• Teachers 

• Principals 

• Superintendents 

• School board members 

Think of each employee as a school ambassador 
to the community. Provide them with communi- 
cation skills and training. They are among the 
district's most important messengers. 
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Working with the Media: Making Sure Your Message Is Heard 

Think back to the last interview you did with a reporter. How did what you said during the 
interview translate to the printed page or television? Was your message heard? Working with the 
media can be challenging, especially when you are trying to communicate complex data and 
information that doesn't neatly fit into a 10-second sound bite. 

Before any media interview, know the opposing side's point of view as well as you know your 
own. Decide what the goals are for the interview. For example, do you want to raise awareness 
about trend data that reveal improvement in student test scores? Are you sharing district strate- 



The public schools really are the public's schools . If they are given 
the impression that they are welcome to participate only if they 
can do something educators think worthwhile , this puts the cart 
before the horse (i.e., treats citizens as a means) and disconnects 
the public from schools . 

— David Mathews , \ 
Is There a Public for Public Schools?, 1996 • 



gies for turning around low-perform- 
ing schools? Identify your audiences 
and which messages resonate best 
with them. Stay positive, succinct, 
solution-driven and in control of the 
interview. 

For more on media interview tech- 
niques and related materials, see 
Appendix B on page 54. 




Key points in Chapter 3 include: 

o Teachers, principals, district staff and the community should be involved in gathering, 
analyzing and discussing data. 

o Success in promoting data-driven school improvement depends on educating parents 
and community members about what information different data convey. 

o Understanding what data the community uses to measure whether schools are improv- 
ing helps superintendents avoid major disconnects in communicating information 
with taxpayers, business leaders, parents and other community members. 

o The most effective strategy for listening to community concerns is a face-to-face 

meeting with different or multiple constituencies, such as a focus group, a community 
forum, a study circle or a superintendent leadership council. 

o Crafting a comprehensive strategic communications plan helps superintendents 
identify key messages, audiences, results, effective tools (e.g., web-based data charts) 
and timelines. 
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• How can the school district's culture shift to one that encourages using data 
to drive decisions? 

• What roles should a superintendent, school board members, central office 
staff, principals, teachers and others play in collecting, processing, 
interpreting and using data? 

• How will key players be trained to effectively participate in 
data-driven decision-making? 

• Is the data system being built able to foster continuous improvement in 
the school district? 

• What can be learned from other successful districts? 



C apitalizing on recent advances in technology and research regarding educational improve- 
ment, superintendents nationwide have developed countless strategies for using data to 
drive decisions at the district and school levels. Whether they are data experts or novices, 
however, they probably could be doing more to make this invaluable resource — data — work 
for them. 



How? Listen to the stories of superintendents and other education leaders from across the 
country who have been using data to drive decision-making. More than anything else, their 
experiences underscore the importance of team build- 
ing. After all, leaders are only as good as their teams. : 



Superintendents illustrate the power of building a 
data-friendly culture, ensuring that school board 
members and staff understand their roles and respon- 
sibilities, providing the training needed to foster new 
data skills and establishing a system focused on 
continuous improvement. 



This is the critical point of data: It's not necessarily 
the answers that are most important; it's the 
discussion that occurs and the questions that are 
asked because of the data. 

— Karen Bates , superintendent , 
Lake Washington (Wash.) School District 



Changing the Culture 

Data traditionally have not been a major factor in the ways schools and districts make 
decisions. The intuition of principals and teachers, advocacy by parents and political interests 

often have guided decision-making. But all that is changing. 

Q 
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The call for data-based accountability, trumpeted first by state education leaders, has been 
met with skepticism by many of the people in schools who are being asked to collect and use 
data. Teachers and principals sometimes react initially with fear, distrust and resistance. Their 
first questions, if they dare to ask, concern how the data will be used against them. 

“Teachers are afraid of data because they think their jobs may hang in the balance, based on 
how their students are doing. Principals are worried about where they will get the time to ana- 
lyze data/' says Superintendent Ray Yeagley, who has spent years easing the data-driven fears of 
educators in the Rochester (N.H.) School District. "The superintendent has to help people see 
the global picture about what data can and can't do. I need to help my principals find the time 
to make this a priority. I need to reassure my teachers about how it will and won't be used, so 
they feel it won't be used unfairly." 

To overcome these hurdles, education leaders recommend: 

o Taking every opportunity to show principals and teachers that data are not being 
used to “get" them, but to improve student learning. Tom Glenn, superintendent of 
the Leander (Texas) Independent School District, says he used every outlet he could 
find — including the district newsletter, meetings, workshops and private conversations 
— to get that message out. “We still encounter resistance, from time to time, to laying 
the data on the table. But that happens less and less as our culture helps people under- 
stand that we're not asking these questions to harm people," he explains. “You have to 
act appropriately with the data and make sure people understand: We're working on 
the system." 

o Helping district faculty and staff understand that using data, much of which already is 
public, is a shared responsibility. The Poway (Calif.) Unified School District has adopted 
the philosophy that it is made up of “professional learning communities," says Ray 
Wilson, director of learning support services. “We all work together for the common 
good. If one of us has poor test scores, it's our job as a department to figure out why and 
help that person get test scores up," he says. To help people become more comfortable 
and familiar with data, Wilson's department devised a scavenger hunt for the district's 

part-time school-based data staff. “They 
had to create a school profile that included 
teacher credentialling rates, mobility rates, 
test data and the number of years that the 
teachers at each grade level had been 
teaching," he says. “All this information is 
available on the Internet. Many people 
were surprised how much information 
about their school, which they struggle 
every year to collect, is public." 



| The importance of data for administrators, policymakers and 
\ teachers in the classroom — to be able to break data down and 
j know where strengths and weaknesses reside — is crucial if 
! you want to make any kind of improvement. 
j — Monte Moses, superintendent, 

! Cherry Creek (Colo.) School District 



Chapter 4: Strategies for Success 



Collecting the Right Kinds of Data 

Leander (Texas) Independent School District 

The need for data has been felt acutely, even 
among school systems in Texas where the Texas 
Assessment of Academic Skills (TAAS) has set a 
national standard for data collection, says Tom 
Glenn, superintendent of the Leander (Texas) 
Independent School District. 

In the early 1 990s, as TAAS was taking hold, 
Glenn and other district officials realized that their 
teachers and principals needed something the state 
test did not provide: frequent updates on student 
progress that could be used to guide instruction 
and address problems throughout the school year. 

Despite its strengths, TAAS tests students only 
once annually, near the end of each school year. 
Results are reported in the summer, when educa- 
tors are powerless to address achievement short- 
falls. What those shortfalls might be remains some- 
what murky, because TAAS reports on student per- 
formance in terms of broad categories rather than 
specific skills. 

So Leander began exploring an additional test- 
ing option. Principals and teachers worked togeth- 
er for two years to revise language arts and mathe- 
matics curricula, then worked for another year to 
help develop a test based on those curricula. 

The resulting Student Profile System tests stu- 
dents from kindergarten through high school in 
language arts and mathematics three times a year 
— at the start of the school year, in the middle and 
at the end. Principals, teachers and students receive 
reports, including easy-to-understand charts that 
show specifically where students excel and where 
they need to focus more attention. 

"The first time we used the test, on the first 
day of the school year, we had elementary students 
in tears," recalls Glenn. 'They said, Tm in the 
fourth grade, and I'm taking the fourth-grade test 



at the beginning of the year — I don't know any of 
this stuff!' It took us a while to help students under- 
stand that they were not expected to know any of 
this stuff yet — that we were just trying to find out 
how much of it they did know, so we didn't waste 
time at the beginning of the year teaching them 
things they already knew." 

Teachers and principals were concerned, too. 
"We promised them: 'We will never use this assess- 
ment to evaluate you; it is for your use,'" Glenn 
says. "We had to prove it over and over again. We 
told people that the system is the problem; individ- 
ual people are not necessarily the problem. We 
called it 'failing forward' or 'learning from our mis- 
takes' or 'working on the system' — we called it 
lots of different things. We had to help people trust 
and understand that, as we collected data and 
found things that weren't always positive, they 
were not going to be hung out to dry." 

Glenn says the 1 3,200-student district contin- 
ues to use the system because it: 

• focuses attention on learning, not testing; 

• generates data that can be used to 
improve education, not just categorize 
students and schools; and 

• allows students, as well as teachers, to 
manage the learning process better. 

Principals, teachers and students report that 
the system is helping them work more effectively 
than ever before to improve learning — which is 
the purpose of testing, Glenn notes. "Once people 
understand that you will not use data to beat them 
up," he says, "then they will help you collect that 
data and be very honest about it." 
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Ultimately, to shift the culture of your district to not only tolerate but embrace data, you 
must not rely on data so much that you discount the training, experience and expertise of the 
people who you want to use the data, says David Benson, superintendent of Blue Valley (Kan.) 
School District. 

In the early stages of Blue Valley's data-collection efforts, a respected administrator, now a 
deputy superintendent, fumed visibly for several weeks before finding the words to voice his 
frustration, Benson recalls. "He came out of a meeting and said, 'Dave, you're ignoring my intu- 
ition. I feel this way about this — and because I feel this way, you have to listen to me/" Benson 
says, "It was such an epiphany for me to understand the importance of that point: You cannot 
take a human institution and just crunch it to data points. You have to listen to people." 

Benson listened to the administrator, as well as to 
other critics. As a result, the district's data system has 
been improved. Moreover, it has survived and thrived 
as principals and teachers throughout the district have 
made way for data in their work. Today, the 17,000- 
student district uses local, state and national test data 
in its accreditation process, school profiles and strate- 
gic planning. But that did not happen overnight. 

"It's a gradual process of proof," says Benson. "If 
you can demonstrate that what you're doing can 
make a difference in the lives of the students we serve, then it will gain acceptance. If it is data 
for data's sake only, and is not related to our primary mission of providing an academic educa- 
tion to children in a safe and orderly climate, then it will not gain acceptance." 



One of the primary responsibilities of a leader is to 
help people confront problems. The way you confront 
problems is you identify them , and the way you iden- 
tify problems is by looking at data. 

— Terry Grier, superintendent, 
Guilford (N.C.) County Schools 



A Closer Look 

Spence Korte 

Superintendent , Milwaukee (Wis.) Public Schools 

Why do school districts need to be able to make decisions based on data today? 

We have a history of going after money for a hot new project and running it for three or four 
years, and then it becomes part of what we do. Nobody is really getting at the critical ques- 
tions: We just spent over a billion dollars this year — is that money put in the right places? Is 
it getting results? 

In Milwaukee, we've been building a sophisticated network to collect student informa- 
tion. Each of our schools gets a report on how their students are doing. It's pretty practical 
from a principal's point of view. For example, how many ninth graders are coming in below 
grade level, and do I need to change my hiring pattern so I have more remediation for kids 
coming into my school? The information is online. People in schools can access all their 
records 24 hours a day. 
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What has worked well for your district in terms of setting up a sound data system? 

Driving our whole technology design and infrastructure is an ex-high school principal. He 
comes with 24 years of field experience in knowing what information has to look like to add 
value for schools. Instead of recruiting an information systems technician to manage this, 
we've gone the other way and drafted a practitioner. The result is that our data tend to be 
more user-friendly and more school-oriented than they might be otherwise. That's something 
to think about when you're out looking for an IT person. You can always surround that per- 
son with technicians, but if he or she doesn't know what schools need, the fit you get might 
not be what you want. 

How important is data training for teachers , principals and 
other district employees? 

It's absolutely critical. In retrospect, one of the things 
that we did right was take an existing building and set 
it up as a training center. We cycle people through 
every day. 

All our best technology teachers are there. Almost 
all of them have a background in schools, so it isn't 
"techspeak" as much as it is, "When I was a teacher at 
such-and-such a school, this was a big problem for me. 

I bet you have that problem at your school. Let me show you how we solved it with technolo- 
gy." We have good credibility with the teachers. 

We do our own training about 90 percent of the time. Sometimes we'll have a vendor 
contract where they may do the installation and 500 hours of training or something. But we 
don't rely heavily on vendors. In the technology world, the vendor can be on the top of the 
heap at the beginning of the year, and out of business at the end of the year. We want to fig- 
ure out how we can institutionalize technology as a major part of the way we do business. 

How do you accomplish that? Who needs to be involved in the process? 

You need to start with the end users, and they need to be educated about what the district- 
level folks need to make those decisions. You've got to get buy-in at each of those points 
along the way. 

You always go through that part where you're running the old system and you have a 
new system ready to go. Ten percent of the people are ready to dump the old system and go 
to the new system. You need to find a way to make it politically palatable to say, by a certain 
date, "We can't afford to run two systems. We don't have the people for it. How can we help 
you get to where you need to be?" It really does take a driver somewhere in the organization 
who understands that the discomfort quickly gives way to comfort if you do it well. 



Businesses don't keep data that's useless , that 
doesn't inform them of anything, yet in education 
we have data that just runs all over us. We have to 
target it and organize it in such a way that it 
informs us so that we can make better decisions. 

— David Benson , superintendent , 
Blue Valley (Kan.) School District 
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How do you get the teachers and principals to buy into data-driven decision-making? 

You have to show how it adds value. And at some point, I just had to say, "I understand your 
problem, but we're going to do this. Get to it." Part of leadership is that, at some point, you 
have to be able to see where it's going and have the courage to stay with it until it actually 
works in all of your schools. A lot of people are not willing to pay the political price. You have 
to be willing. 




The culture of a school district will not smile favorably on data-driven decision-making for long 
if the members of that culture continually step on each others' toes or fumble the ball when it's 
their turn to run with the data. That is why the superintendent must draw the game plan in 
clear, bold strokes — to help board members, district staff, principals and teachers understand 

"It all has to work in coordination," says 
Douglas Otto, superintendent of the Plano (Texas) 
Independent School District. "Everybody has to 
have the same vision of what we're doing and 
why. Everybody has to understand what kinds of 
data are important to collect and how they're 
used. From the school board and superintendent 
on down, there has to be that common sense 
of purpose." 

The 47,000-student district's data efforts appear to be paying off. Plano educators monitor 
student performance on the Iowa Tests of Basic Skills, the Cognitive Abilities Test, the Texas Assess- 
ment of Basic Skills and the Scholastic Aptitude Test. The district also collects and reports data in 
areas such as staffing, spending and enrollment patterns to inform decision-making continually. 

To keep such a data system humming along smoothly, Otto says, team players must under- 
stand their special roles and responsibilities: 

o Superintendent and School Board. They share leadership of the district’s data 

efforts but play significantly different roles. In Plano, the division of labor has been 
clear: The board adopts policies to govern the system; the superintendent manages 
daily operations. "The board made sure we had the appropriate policies in place to 
make sure we were going to collect data, analyze it and use it to improve instruc- 
tion," says Otto. "Board leadership is extremely necessary." 

To make effective policy decisions, the board must understand data. For exam- 
ple, the board needs to know that some data trends are the result of a "common 
cause" (a regular feature of the school system, such as normal student attendance 
patterns) and some are the result of a "special cause" (an unusual circumstance, such 



their important roles in the data system. 



i There are no secrets to success. It is the result of 

\ preparation, hard work and learning from failure. 

i — Colin Powell, ! 

! | 
i U.S. Secretary of State j 
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as a flu bug that causes many students to miss school on a big test day). Otherwise, the 
district might spend an inappropriate amount of time trying to solve a problem that was 
a fluke. 

The superintendent's role, then, is to empower central office and school staff to use 
data as effectively as possible to meet the board's mandates. Max Riley, superintendent of 
the Lawrence Township (N.J.) Public Schools, says the superintendent should send staff a 
clear message: "This is not a 'gotcha.' I'm going to give you every tool I can think of to 
help you succeed." 

0 District Staff. Central office and other 
staff at the district level must be pre- 
pared to discard longstanding ways of 
doing business if data dictate that a 
new approach is needed. In 1996, for 
example, Plano brought in consultants 
to conduct a curriculum audit to com- 
pare district practices with established 
standards. Based on the audit, Plano 
revamped its curriculum management 
team and expanded its research and 
assessment team to help "make wise decisions about what data we're going to collect, 
who's going to collect it, who's going to analyze it, who's going to be trained, and how it 
gets put back into the curriculum and strategies used in the classroom, "says Otto. 

0 School Site Councils. In many districts, school-based management teams — often called 
councils or committees — are taking up data along with the other tools they use to help 
make decisions for their campuses. "We'd had site-based improvement committees in 
place for many years," Otto says, recalling the changes that followed Plano's curriculum 
audit. "All of a sudden, one of their chief roles was to monitor areas of improvement in 
school, make recommendations, work with staff, and use the same data that principals 
and teachers were getting so everyone was on the same wavelength." 

0 Principals and Teachers. Though principals bear ultimate responsibility for what hap- 
pens in their schools, they must work with teachers to use data in ways that best serve 
their unique student populations and school needs. In Plano, principals receive assis- 
tance — and are held accountable — for this work. "Principals receive data training. It's 
not voluntary; it's mandatory. They have to know how to use data on their school-based 
improvement committees. It's part of their performance appraisal," says Otto. "Principals 
also need to play a monitoring role to make sure that teachers are implementing the cur- 
riculum, using effective strategies and improving student achievement." 



Data-driven decision-making is not someplace we've arrived; 
it's still a journey and always will be. But it's a journey 
where people understand how you buy the ticket and get 
on the tain. And if you're not on the tain, you need to 
find another place to be, because this is the way we have 
to operate. 

— Yvonne Katz, superintendent, 
Beaverton (Ore.) School District 
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Yvonne Katz, superintendent of the Beaverton (Ore.) School District, says data must be dis- 
tributed across grade levels, so elementary school teachers know how to prepare their students 
for middle school level work, and high school principals know how well their incoming fresh- 
men will be prepared to perform. "Everyone can see what the data are showing" in Beaverton, 
says Katz. "As you look at groups coming through, you can plan what your focus will need to be 
in future years, when the kids hit your level." 




As school districts become more sophisticated in their use of data, they are starting to look 
beyond their own borders to appreciate the helpful role that state departments of education can 
play. This outlook is reversing years of mistrust and resentment, begun when states started col- 
lecting data that district officials worried would be used against them. As local school systems 
have become more confident and comfortable with data-driven decision-making, they have 
looked to the states for data training and other forms of technical support. 

Katz retains a unique perspective on the state role, based on her experiences as associate com- 
missioner for general education of Texas in the early 1980s. "We had a tremendous training pro- 
gram in Texas, bringing in principals and vice principals to be trained for a whole week in data 
usage, then visiting campuses around the state," she says. 

New York provides a different kind of support for districts. State assessment scores not only 
are distributed widely, but are broken down to allow comparisons among similar school districts, 
such as those that serve high percentages of students eligible for federal free or reduced-price 
lunches. 

"Some fairly helpful comparisons can be made for follow-up for the purposes of instructional 
development for classroom teachers and principals," says Kevin McGuire, who served as a super- 
intendent for 18 years before becoming director of the New York State Center for School Leadership. 

"There are extraordinarily talented educators in every comer of the state. By linking people 
together, we think we can help everyone grow, the adults as well as the children," McGuire says. 
"For example, if people are having difficulty with students who are remedial in nature, they can 
talk to a similar school district that seems to be meeting the needs of similar students in a more 
effective way." 
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Who Does What? 


Data-driven decision-making, especially in 


the early stages, demands that district 


leaders point the way. Superintendents and school boards both must play important 


but distinct roles. 


The superintendent generally: 


The school board generally: 


• Translates the board's vision for the 


• Establishes a vision for the school 


school district into measurable goals 


district based on data showing what 


based on data. 


has been achieved so far and what 


• Works with district faculty, staff, 


progress is necessary. 


parents and other community stake- 


• Spells out — for the superintendent 


holders to craft plans for meeting 


and other employees and stake- 


goals by certain dates. 


holders — how district performance 


• Collects data to show clear, steady 


will be evaluated. 


progress. 


• Reviews relevant data to evaluate 


• Celebrates successes, evaluates 


district progress toward identified 


shortcomings and revises plans for 


goals. 


improvement based on data, along 


• Revises goals and plans for 


with the board. 


improvement based on data. 


Broad participation In improvement efforts serves to: 


• Promote a high level of support for 


• Prepare participants for their role in 


those efforts. 


implementing improvements. 


• Generate sound solutions by 


• Increase ownership of and 


expanding the discussion. 


commitment to specific strategies. 


• Motivate participants and their 


• Empower important stakeholder 


associates. 


groups. 


• Increase the likelihood that the 


• Foster lasting, rather than 


effort will lead to constructive 


temporary, change. 


action. 


Source: At Your Fingertips: Using Everyday Data to Improve Schools, 1998 
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Training the Players 

Superintendents know that data-driven decision-making requires new knowledge and skills. In a 
2000 AASA membership survey, superintendents overwhelmingly called for training on using 
data. They also knew their staffs needed a crash course in collecting and analyzing data, using 
data-processing technology, and operating the decision support system. 

"I can sit up here in the superintendent's office and look at the data and issue directives, but 
then people don't understand what I'm doing or why I'm doing it," says Glenn. "You've got to 
empower people to use data. They'll help you identify problems — and often, through team- 
work and looking at other ways of doing things, they will come up with solutions." 

To help schools use Leander's new Student Profile System, which generates data on student 
progress throughout the year, the district provided teachers with lots of training, much of which 
directly involved them in helping develop the district's student profile test. A specially trained 
"data group" gathered data and reported it back to schools in user-friendly formats. Principals 
received training in helping teachers use data individually, in grade-level teams and as a school. 
"We did a lot of modeling with people," says Glenn. "I met with principals three or four times a 
year on their campuses to talk about their data: What's going on? Where are the peaks? Where 
are the valleys? What are they doing about the valleys?" 

But Glenn does not advocate trying to make a data technician of every school employee. 
More important is training that emphasizes the conceptual underpinnings of data-driven 
decision-making — the core principles of how educators can use data to improve education. 

"I can hire all kinds of people to come in and help us do the grunt work of collecting the 
data, putting it into charts and those kinds of things," says Glenn. "But as many people as possi- 
ble in the system have to understand the philosophical basis of what you're trying to accom- 
plish. Sometimes that's where we fail in education — in not helping people understand what 
we're trying to accomplish with all this paperwork we're asking them to do." 



Depending on what type of training is needed, superintendents might want to explore 
various ways of providing training. Should an outside vendor be hired to train staff, or should 
this be handled internally? Should one person spearhead the effort? Or would it be best to 
build a team? 

Only you can answer such questions for your district. But successful superintendents and 
other education leaders have learned the following lessons: 

o Not all employees need to receive training, but employees at all levels need to learn 
about data-driven decision-making. This helps build buy-in at all levels and ensures 
that the system works the way it should. 
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o Perhaps because data can be intimidating at first, districts often rely too heavily on 
outside consultants. The result can be generic training that is poorly matched to the 
specific needs or problems of a district. If a data effort is to be sustained over the long 
term, the district must cultivate in-house trainers who can help colleagues use the 
local data system. 

0 Actual district data should be used in training only when appropriate. If training focus- 
es on data analysis, then using local data can provide a compelling context for partici- 
pants. But if training is intended to help participants master new data software, local 
data can be a distraction. 

When in doubt, remember: Data also can help tailor training to the unique needs of a dis- 
trict. Consider the technology self-assessment used by teachers in Blue Valley. Teachers complete 
a 43-item questionnaire to determine how they use technology and how they might use it 
better. The district uses that data to design professional development in technology. "Because 
we have used data, we have targeted inservice to specific, identified needs, rather than just using 
broad-brush technology inservice," says Superintendent David Benson. 

The return on investment for data training is incalculable. "No matter whether youTe a huge 
district or a very small district, if teachers are trained and have some knowledge of how to col- 
lect and analyze data, they can do some remarkable things in the classroom — without needing 
some decision from on high passed down telling them what to do," says Otto. 




a 



Continuous 



Increasingly, leading school districts are using data-driven decision-making to ensure continuous 
improvement. They do so by moving beyond the initial work of assessing their own schools' 
performance, comparing those schools against each other and monitoring trends over time. 
These districts look beyond their borders — to benchmark their schools' performance against 
that of other top-performing schools across the country. 



Benchmarking is more than comparing test 
scores. Done well, benchmarking helps dis- 
tricts learn what it takes to be effective, how 
well they are using key strategies, where 
improvements are possible or necessary, and 
how they can learn from the best practices of 
other successful districts. 



! Meaningful school reform is probably more about using 
| data effectively than about any other tool you can use. 



— Douglas Otto , superintendent , I 
Plano (Texas ) Independent School District t 




A recognized leader in district benchmarking practice is the Western States Benchmarking 
Consortium. Founded in 1997, when a group of education leaders from high-achieving districts 
gathered to devise strategies for improvement, the consortium continually has brought together 
superintendents and others from member districts to develop research-based benchmarks for 
district success. 



Using Data to Improve Schools: What's Working 



Looking for Value Added 

Plainville (Conn.) School District 

In the three years he headed the Plainville 
(Conn.) School District, then-Superintendent 
Max Riley raised student scores on Connecticut 
state exams to record levels — but he didn't do 
it alone. Riley recruited a "stat jock" from Central 
Connecticut State University to help principals 
and teachers use data for improvement. 

James Conway, a psychology researcher and 
professor, was contracted by the district to "act 
as our coach for thinking about. data," says Riley. 
Conway combed through reams of available data 
and analyzed performance trends over time and 
differences among schools. Then the researcher 
made himself available for confidential consulta- 
tions with principals about their schools' data. 

"I wanted principals to have access to some- 
one with whom they could have conversations 
without necessarily having to go through the 
superintendent," Riley says. "It was understood 
from square one that these were confidential 
conversations and that Jim was just there to help 
principals in whatever they wanted to do, as 
long as it was focused on data and student 
learning." 

Providing that sense of security made a dif- 
ference. Principals latched onto Conway. They 
called with questions. They explored the data. 
They started seeing their schools in new ways. 

For example, most people in Plainville tradi- 
tionally had focused on raw test scores — and 
ignored relevant data about the student popula- 
tion — in evaluating the success of the district's 
three elementary schools. The school that served 
mainly children of white-collar parents routinely 
earned top marks. Two other schools, both of 
which served children of comparatively low- 
income families, fared less well. 



But Conway toppled those perceptions with 
a value-added analysis. Essentially, Conway col- 
lected data on how well students performed 
when they entered each school, then examined 
how much progress that school enabled students 
to make. According to this analysis, the two 
schools with the lowest test scores added value 
to children's education at double the rate of the 
school with the highest test scores. 

"Our highest-performing school was under- 
performing in terms of moving kids along," says 
Riley. "It sent shock waves through the organiza- 
tion. It turned everything upside-down in terms 
of people's mythological understanding about 
who's doing a good job." The value-added 
analysis also helped the principals at all three 
schools better understand where they were pro- 
viding effective instruction and where they need- 
ed to work harder, he adds. 

Now superintendent of the Lawrence 
Township (N.J.) Public Schools, Riley is setting up 
a similar system. Working with a researcher from 
a nearby university, Riley encourages principals to 
sift through data to see how they might increase 
their value added for students. 

Working with college and university 
researchers can be particularly advantageous for 
relatively small districts like Plainville and 
Lawrence because districts can pay relatively little 
and get data services tailor-made to meet their 
needs. "Of course, the board loves it, because 
they feel like they're making focused, informed 
decisions about how to spend the money," says 
Riley. "Data also give principals the ammunition 
to come in at budget time and say, 'Look, it's 
your own darn data system showing where I 
need help. Now, I need help.'" 
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Field-testing those benchmarks during the 2000-01 school year, the consortium's seven mem- 
ber districts conducted self-assessments and shared results and insights. Now, member 
districts are using the benchmarks in everything they do, from setting goals and strategic 
planning to designing professional development and establishing accountability systems. 

"In Plano, we are developing strategies for our school-based improvement committees to use 
the benchmarks to drive improvements in buildings," says Otto, whose district has been at the 
forefront of the consortium's work. "One of our weaknesses is that we're still not at the point 
where every teacher is using the data to improve instruction. We know that from our own obser- 
vation and from talking with groups of principals and curriculum coordinators. That tells us that 
more training is needed." 

How does it work? Based on recent research showing links between specific organizational 
policies and student achievement increases, the consortium has adopted four strategic areas of 
focus for benchmarking: student learning, capacity development, learning community develop- 
ment and data-driven decision-making. 

Under each benchmark, a district's performance shows it to be at one of four stages of devel- 
opment: emergent, islands, integrated or exemplary. For example, the consortium has developed 
a data-driven decision-making benchmark for "relating investments, outcomes and improve- 
ment strategies." A district with "emergent" performance in this area collects data that are not 
maintained consistently over time and are not linked well enough to key outcomes to inform 
critical decisions. By contrast, the data needs of an "exemplary" district are clearly articulated 
and met in all stages of planning and implementation. 

For each stage of development, the benchmarks include "behavioral indicators" (beliefs and 
actions evident in the system) and "support indicators" (things that support desired behaviors, 
such as professional development, policies and resources). These indicators not only let districts 
know how well they are doing, but also give them sound, research-proven ideas about how they 
can do better. 

The process of developing the consortium's benchmarks has been more valuable, in some 
ways, than the benchmarks themselves. "It really generates a lot of professional opportunities," 
says Otto. "Having superintendents sitting around the table talking about these benchmarks — 
what we're doing and the strategies we're employing — really raises the level of conversation 
with regard to school improvement." 

"The consortium holds great promise for us in terms of continuous improvement based 
on objective standards," agrees Benson, whose Blue Valley School District also participates in 
benchmarking. "You want to make continuous improvement in those areas where you can say 
objectively: 'This will make a difference for kids.'" 



O 

TRJC 



ii** 



55 



49 



Using Data to Improve Schools: What's Working 



Benchmarks for School District Success 

The Western States Benchmarking Consortium's benchmarks for school district performance allow a district's 
performance to be judged to be at the "emergent/' "islands/' "integrated" or "exemplary" stage of development. 

The table below describes indicators at different levels of performance for how well educators use data to personalize 
education or students and profile and monitor student performance. 



Data-Driven Decision-Making: Using Information to Improve Instructional Practice 


Emergent 


Islands 


Integrated 


Exemplary 


Data for 


Data for 


Data for 


Data for 


Personalization 


Personalization 


Personalization 


Personalization 


The instructional 


Some schools provide 


Evidence shows that the 


Teachers routinely use 


approach is "one-size- 


examples of new 


programmatic use of 


performance data to truly 


fits-all/' with underper- 


approaches to accelerat- 


data in most areas such 


personalize learning for 


forming and accelerated 


ed or extended learning 


as accelerated and 


all students. 


students "pulled out." As 


that are driven directly 


extended learning pro- 




a result, there is little or 


by examination of data. 


grams flows directly 




no integration among 




from examinations of 




remedial, accelerated 
and ongoing instruction. 
The concept of personal- 
ized instruction for all 
students based upon 
individualized perform- 
ance data is foreign. 




performance data. 




Profiling and 


Profiling and 


Profiling and 


Profiling and 


Monitoring Student 


Monitoring Student 


Monitoring Student 


Monitoring Student 


Performance 


Performance 


Performance 


Performance 


The district has not 


The student perform- 


The district provides 


Data are aggregated/ 


organized a formal 


ance database is stil! 


a profile of student 


disaggregated to class- 


student performance 


nonexistent, but more 


performance for each 


room/school/district levels 


database. As a result, 


people are demanding 


school. All schools use 


to determine necessary 


schools are left to their 


that it be designed and 


these profiles to set 


improvements in instruc- 


own devices to track 


implemented to improve 


annual and longer 


tional practice. Heavy 


performance over time. 


curriculum and instruc- 


range improvement 


evidence of reliance on 


Teachers have great 


tion. Schools are strug- 


targets. Student progress 


student performance data 


difficulty in determining 


gling with the inadequa- 


reporting is based on a 


from multiple sources 


the peform-ance level 


cy of the performance 


continuum of progress 


exists. The system main- 


of students who move 


information base regard- 


in achieving well-defined 


tains a performance profile 


from school to school. 


ing timeliness, validity. 


content and perform- 


for every student, integrat- 


Traditional grade-based 


reliability and compre- 


ance standards. More 


ing all information about 


report cards provide 


hensiveness. The formal 


schools are providing 


the student's performance 


parents with information 


report card is still grade- 


parents with specific 


over time. Parents receive 


on student progress. 


based and remains the 


suggestions regarding 


frequent progress reports 




primary mode of com- 


techniques for family 


of growth on the standards 




munication with parents 


reinforcement of 


continuum, accompanied 




regarding student 


these standards. 


by suggestions for home 




progress. 




reinforcement. 


Source: Western States Benchmarking Consortium 
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The following school district performance benchmarks developed by the Western States 
Benchmarking Consortium describe indicators of district success in four major areas of 
educational performance. 

Student Learning 

o Ensuring learning for all students 

o Integrating standards 

o Incorporating innovative practice 

0 Integrating technology 

© Developing a coherent curriculum 

Capacity Development 
° Expanding organizational effectiveness 

0 Adopting a curriculum management system 
0 Promoting innovation 

° Improving professional/organizational learning 

Learning Community Development 
° Understanding and using state academic results 
° Providing community-based learning opportunities 
° Building community partnerships 

° Building community development 

Data-Driven Decision-Making 
° Using a variety of data effectively 

° Using information to improve instructional practice 
° Using data to affect student performance 
0 Relating investments, outcomes and improvement strategies 
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Danger Zones 

Superintendents charging into the unfamiliar territory of data-driven decision-making 

sometimes see the edge of the cliff only just before they reach it. Here are some lessons learned 

by district leaders who have been there: 

© Don't rush headlong down the path to collect data and enforce accountability before 
bringing on board all the people needed to make the new system work. Talk. Listen. 
Build consensus about what is most important to do. Otherwise, you might find your- 
self out in front — with no one behind you. 

o Don't fish blindly for meaning in data. If you do not know what questions need to 
be asked — before you even collect the data — you are not likely to find the answers 
you need. 

© Don't bombard people with data for its own sake. Don't report it just because it is 
available. Stick with data points that can drive decisions for improvement. 

° Don't lose track of what is most important — such as key indicators of student learn- 
ing progress. If you place equal priority on many data points, some of which might 
have little to do with your district's core mission, then your team will not know where 
to focus its energy. 

© Don't forget to draw the "big picture" for the people collecting, interpreting and using 
data. They need to understand why they are being asked to do this and what it all 
means. Explain it again and again and again. 




Key points in Chapter 4 include: 

® Successful data-driven decision-making requires a shift in the culture of a school 
district that encourages the use and analysis of data without fear of reprisal. 

q The superintendent, school board members, principals and teachers must be clear on 
the purposes and goals of data collection and analysis. 

« To help educators use data effectively, it is important to provide training and 

assistance, because data-driven decision-making requires new knowledge and skills. 

© School districts using data-driven decision-making ensure continuous improvement by 
benchmarking their schools' performance against that of other top-performing schools 
across the country; benchmarking helps districts learn what it takes to be effective. 



Parents ; taxpayers and educators were asked to rate the following items in importance for 
school accountability. 



Category Parents Taxpayers Educators 



School safety 


9.6 


9.4 


9.3 


Teacher qualifications 


9.3 


9.2 


8.3 


Class size 


8.9 


7.9 


8.8 


Graduation rates 


8.7 


8.2 


8.3 


Dropout rates 


8.3 


8.1 


7.4 


Statewide test scores 


8.2 


8.0 


7.1 


Parental satisfaction 
survey data 


8.1 


8.0 


7.0 


SAT/ACT scores 


8.1 


7.9 


6.9 


Percent of students 
promoted to next grade 


8.0 


8.1 


7.0 


Course offerings 


7.8 


7.9 


7.3 


Attendance rates 


7.8 


8.0 


7.6 


Per-pupil spending 


7.6 


7.6 


8.0 


Student satisfaction 
survey data 


7.5 


7.0 


7.1 


Teacher salaries 


7.3 


7.8 


7.6 


Hours of homework per week 


7.2 


7.3 


6.3 


Number of students 


7.2 


7.2 


6.7 


Percent of students who go 
on to a four-year college 


7.0 


6.9 


6.8 


Percent of students with 
an "A" or a "B" average 


7.0 


6.5 


5.8 


Number of students 
per computer 


6.9 


6.4 


6.1 


Percent of parents who attend 
parent-teacher conferences 


6.4 


6.6 


6.3 


Demographics of students 


4.5 


4.6 


5.0 



Source: Reporting Results: What the Public Wants to Know, 1999 
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Working with the Media 



Media Dos and Don'ts on Data 



0 Do make sure your district has a comprehensive communications plan that addresses 
the release of potentially controversial data such as test scores, dropout rates, teacher 
quality ratings, school safety reports and other indicators. 

o Don't wait until the day test scores are released to deepen the media's understanding 
about what the tests measure, how the district will use the test scores and what the 
results tell parents and community members. This should be an ongoing process. Be 
proactive. Keep the media informed. Make an occasional courtesy call even when you 
don't have to. 

0 Do make sure staff is apprised of test results and other sensitive data before the media, 
so everyone is focused on the same messages. For example, if the state releases test 
scores one day before the scores are released publicly, hold meetings with teachers, 
principals and other district staff to inform them of the results. 

0 Don't hide data from reporters. It's public information. 

0 Do talk to the reporter first when inaccuracies surface in a story, such as factually incor- 
rect data or scores. Make sure you have documentation to back up your claim. If prob- 
lems continue, contact the reporter's editor. 

0 Don't neglect to share what is working in the district. Suggest feature stories that show 
trends in school improvement backed by data. 



Interview Techniques: 

How to Handle Tough Media Interviews on Data 



Needling 

Reporter: 

Superintendent: 


"Oh, come on now, you don't believe trend data show improvements 
in teacher quality do you?" 

"Yes, I do, Beth, and... Go on to your key message. 


Twisted Facts 

Reporter: 

Superintendent: 


"So your test scores are down again." (When, in fact, the test has 
changed or a new population of students is tested.) 

"Let's be clear about what the data tell us... 


False Assumption 

Reporter: 


"So I assume you are going to spend taxpayer money to purchase a 
new test since this one shows bad results... ." 


Superintendent: 


"Well, Mike, I wouldn't agree with your conclusion." Steer the 
interview back on message. 


Baiting 

Reporter: 

Superintendent: 


"So why does this community keep attacking your initiatives?" 

"You'll have to talk directly to those who raise concerns. What I can 
tell you is that our district is dedicated to... ." (Discuss public 
engagement efforts to gather and respond to community input.) 


Pregnant Pause 

Reporter: 

Superintendent: 


The reporter remains silent after you answer a question. 

Do not rush to fill in the silence. Wait for the reporter to follow-up or 
ask, "Does that answer your question?" 


Loaded Question 

Reporter: 

Superintendent: 


"You don't deny that the district knew all along about sagging test 
scores and refused to do anything about it." 

"Here's what we have done to improve low-performing schools... . 
Here's what has worked and where we need to improve... ." 



Adapted from a Michigan School Boards Association media slide presentation. 
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Appendix C : Resources 



Reports 

"Building an Automated Student Record System." National Forum on Education Statistics, 2000. 

This booklet is designed to lead education organization decision-makers through the process of 
making the best and most cost-effective decisions about information management systems devoted 
to individual student records. Available online at http://nces.ed.gov/pubs2000/building/ 



"Evaluation of California's Standards-Based Accountability System." WestEd, November 1999. 

This evaluation examines the status and impact of local accountability systems, content stan- 
dards, assessment measures, use of data, consequences and incentives, and challenges and assis- 
tance in school districts across the state. The document reports the findings of the study and its 
implications for policy. Available online at http://web.wested.org/online _pubs/ 'accountability/ 

"Reporting Results: What the Public Wants to Know." A companion report to Education Week’s 
Quality Counts '99. 

A report of what parents, taxpayers and educators think they need to know about schools in order 
to hold them accountable. Available online at 
http://www.edweek.org/sreports/qc99/opinion/edweekresults.pdf 

"Student Data Handbook for Elementary, Secondary, and Early Childhood Education." National 
Center for Education Statistics, 2000. 

This handbook offers guidelines, developed by the U.S. Department of Education's National 
Center for Education Statistics, for current and consistent terms, definitions and classification codes 
to maintain, collect, report and exchange comparable information about students. Available online 
at http://nces.ed.gov/ 

"Using Data for School Improvement: Report on the Second Practitioners' Conference for 
Annenberg Challenge Sites." Annenberg Institute for School Reform at Brown University Tools for 
Accountability Project, 1998. 

This report, developed after the second practitioners conference for Annenberg Challenge Sites, 
provides valuable examples of data collection, ways of thinking about accountability and resources. 
Real-life examples of schools implementing data-collection efforts to improve student achievement 
appear throughout the report. Available online at http.//www.annenberginstitiite.orgAmages/iising_data4.pdf 
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"Vermont Department of Education. Equity and Excellence Action Planning Guide " 
http://vmyNStateytiisleduclactplanlapcover.htm 

This resource provides a look at each step of the action planning process for educators. 



Articles 

"Data-Driven Decisions." Articles by Theodore B. Creighton, Raymond Yeagley, Philip A. Streifer, 
Francis Barnes, Marilyn Miller and George A. Goens. The School Administrator ; April 2001. 

This issue of The School Administator includes a number of articles on data-driven decision-mak- 
ing. Available online at http://www.aasa.org 



"On the Job: Data Analysts Focus School Improvement Efforts." Joellen Killian and 
G. Thomas Bellamy. Journal of Staff Development, Winter 2000. 

This article presents one school district's model for building-level disaggregation and data analy- 
sis. Available online at http://www.nsdc.org/library/jsd/killion211.html 



"Smart Moves: Achieving Your Vision Depends on Follow-Through." Joan Richardson. Tools for 
Schools, National Staff Development Council, August/September 2000. 

This article discusses how school districts move beyond publishing and publicizing a school's 
newly developed vision statement and begin the real work of using the vision as a guide and a 
measure for school improvement. Available online at 
http://www.nsdc.org/library/tools/tools9-00rich.html 



"This Goes on Your Permanent Record." Stewart Deck. CIO Magazine, Nov. 1, 2000. 

This article examines how various school districts are building and implementing data 
warehouses to support districtwide, building-and classroom-level decision-making. 
Available online at http://www.cio.com/archive/110100/permanent.html 



"Using Data to Improve Instruction." Articles by Nancy Protheroe; Jay Feldman and Rosann 
Tung; Yi Du and Larry Fuglesten; Thomas Glass; Brett Geis, Terrell Donicht and Steven Smith; and 
Thomas Fowler-Finn. ERS Spectrum, Summer 2001. 

This issue of the ERS Spectrum includes several articles about the use of data in school 
improvement efforts. 
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Books 

Accountability Dialogues: School Communities Creating Demand from Within. 

Kate Jamentz. WestEd, 2001. 

This book explains some of the common misperceptions about school accountability and 
provides a strong rationale for including accountability dialogues in any accountability system. 
Examples from schools that use accountability dialogues provide a real sense of what can 
happen when responsibility for school improvement is shared among all the stakeholders in 
a school community. 

At Your Fingertips — Using Everyday Data to Improve Schools. Karen Levesque, Denise Bradby, 
Kristi Rossi and Peter Teitelbaum. MPR Associates Inc. and American Association of School 
Administrators and National Center for Research in Vocational Education, 1998. 

This handbook teaches educators new and productive ways of using data. Through step-by- 
step instruction, this book focuses on using available data to improve teaching and learning. 
Information available online at http://www.mprinc.com 



Effectively Using Data to Drill Down and Make Better Educational Decisions. 

Philip A. Streifer. Scarecrow Press, Inc., 2002. 

This book is a guide on how to use data in decision-making, with a strong emphasis on 
asking the right questions in a cyclical process designed to narrow the focus of inquiry. 



Data Analysis for Comprehensive Schoolwide Improvement. Victoria L. Bernhardt. 

Eye on Education, Larchmont, N.Y., 1998. 

This book teaches the layperson how to gather, interpret and use data to improve schools. 
Educators are given practical tools so they can make better data-based decisions. 



Getting Excited About Data: How to Combine People , Passion and Proof. Edie L. Holcomb. Corwin 
Press, Thousand Oaks, Calif., 1999. 

This book outlines a process for showing how well a school or district meets its primary goal: 
sustained student learning. The author offers tips on finding answers to questions about data, 
such as: What data do we need and how do we collect it? 



Improving School Board Decision-Making: The Data Connection. National School Boards 
Foundation, 2001 

This guidebook helps school board members understand what data are, ask for the appropri- 
ate data to inform decision-making, work with the superintendent to understand what data 
reveal, use data to support decisions on policy and budgets and inform the community. 



Is There a Public for Public Schools? David F. Mathews. Kettering Foundation Press, 1997. 

This book provides school leaders with insights for engaging the public when undertaking 
school improvement efforts. 

Thinking About Tests and Testing: A Short Primer in Assessment Literacy. Gerald Bracey. 

American Youth Policy Forum, Washington, D.C., 2000. 

This book is a nonpartisan, objective discussion that helps readers understand the arguments 
now raging about "high-stakes tests" and their consequences. The book is simple and straightfor- 
ward. Available online at http://www.aypf.org/BraceyRep.pdf 



The School Portfolio: A Comprehensive Framework for School Improvement , Second Edition. 

Victoria L. Bernhardt. Education for the Future Initiative, 1999. 

This book shows how to develop a school portfolio tailored to a particular school and vision. 
It explains that school portfolio is the most effective way to ensure a school's success at systemic 
reform. Extensively tested, it is a nonthreatening self-assessment tool that exhibits a school's 
goals, achievements and vision for improvement. 



Schools and Data: The Educator's Guide to Using Data to Improve Decision Making. Theodore B. 
Creighton. Corwin Press , Thousand Oaks , Calif , 2000. 

This book focuses on the relevance of statistics in the day-to-day lives of principals and teach- 
ers. This is an essential resource for any educator who wants to break through the 
statistical confusion to improve skills in problem analysis, program evaluation, data-driven 
decision-making and report preparation. 



Using Data / Getting Results: A Practical Guide for School Improvement in Mathematics and Science. 
Nancy Love. Christopher-Gordon Publishers, 2002. 

This book is a guide for inquiry into improving mathematics and science teaching and 
learning. It examines ways to engage school communities and produce powerful learning. 



Internet Resources 



Center for Accountability Solutions, American Association of School Administrators. 
http://www.aasa.org/data 

AASA has created this website to help school leaders gather, use and report meaningful data 
on student, school and district performance. It includes a comprehensive listing of data 
resources that is updated regularly. 
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Appendix Do Glossary 



Accountability — A process by which educators are held responsible for performance or 
outcomes. 

Application service provider (ASP) — A third-party entity (generally for-profit) that manages 
and distributes software-based services and solutions to customers across a wide area network 
from one or more off-site locations. 



B 

Benchmark — A standard by which something can be measured or judged. Benchmarks can 
be used to monitor progress toward meeting goals. 

c 

Cohort - A group of students or others of the same age moving through a system together 
(e.g., all of the students who enter kindergarten the same year are part of the same cohort). 
Cohorts typically are used to measure progress over time and compare that progress with 
other cohorts. 

Common cause — A regular feature of the school system (e.g., normal student attendance 
patterns). 

Comparative analysis report — A report that contrasts two or more districts, students or 
other groups that have similar characteristics and can be compared to each other. 

Criterion-referenced test (CRT) — A test that measures an individual's performance against a 
well-specified set of standards (distinguished from tests that compare students in relation to 
the performance of other students, known as norm-referenced tests). 

Cut score — A specified point on a score scale, such that scores at or above that point are 
interpreted or acted on differently from scores below that point. In standards-based assess- 
ments, cut scores typically delineate passing from failing, proficiency from mastery, and so on. 

© 

Data — Factual information (such as measurements or statistics) used as a basis for reasoning, 
discussion or calculation. Data can be qualitative or quantitative. Good data must be both 
reliable and valid. 
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Data mining — The analysis of data for relationships that previously have not been discovered. 

Data warehouse — A central repository for all or significant parts of the data that a system 
collects. A data warehouse typically is a database or collection of databases existing in virtual, 
rather than physical, space. 

Decision support system (DSS) — A computer program application that analyzes and presents 
data so users can make decisions more easily. 

Demographic data — Data that focus on the gender, socioeconomic background, race and 
ethnicity of students in a school or district. 

Disaggregated data — Data broken down by specific student subgroups, such as current 
grade, race, previous achievements, gender, socioeconomic status, and so on. 

Drill down process — A method of disaggregating data that begins with a general question, 
followed by increasingly specific questions that focus on smaller subsets of data. 

IF 

Formative assessment — Assessment in which learning is measured at several points during a 
teaching/learning phase, with the primary intention of obtaining information to guide further 
teaching or learning steps. Formative assessments take a variety of forms, including question- 
ing, comment on a presentation or interviewing. 



D 

International test — A test administered uniformly in a number of countries that compares 
student performance country by country (e.g., Third International Mathematics and Science 
Study). 

o. 

Longitudinal data — Data measured consistently from year to year to track progress, growth 
and change over time. True longitudinal studies eliminate any students who were not present 
and tested in each of the years of the study. 



IMl 

Mission — A statement that defines what an organization is created to do and reflects its core 
values and beliefs to guide it toward its goals. 
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N 

Normal curve equivalent (NCE) — Normalized standard scores on an equal interval scale 
from 1 to 99, with a mean of 50 and a standard deviation of about 21. 

Norm-referenced test (NRT) — A test that compares individual student performance with a 
larger group, usually a national sample representing a diverse cross-section of students, (e.g., 
Iowa Tests of Basic Skills, Stanford Achievement Test). NRT results typically are measured in 
percentile ranks. (Norm-referenced tests should be contrasted with criterion-referenced tests, 
which measure performance compared to a standard or benchmark.) 



p 

Percentile score — A score that designates what percent of a norm-referenced group earned 
raw scores lower than a particular score. Percentiles often are divided into quartiles (groupings 
of 25 percent). 

Perception data — Data that inform educators about parent, student and staff perceptions 
about the learning environment, which could also reveal areas in need of improvement. 

Performance test — A test that requires students to demonstrate their abilities by providing 
examples of their work (e.g., portfolios, presentations, experiments). 

Portfolio assessments — Assessments of a collection of a student's educational work that 
shows growth, self-reflection and achievement over time. 

Q 

Qualitative data — Data based on information gathered from one-on-one interviews, focus 
groups or general observations over time (as opposed to quantitative data). 

Quantifiable proof — Proof that can be precisely measured. 

Quantitative data — Data based on "hard numbers" such as enrollment figures, dropout rates 
and test scores (as opposed to qualitative data). 



u 

Reliability — The consistency of test scores over different test administrations, multiple raters 
or different test questions. Reliability answers the question "how likely is it that a student 
would obtain the same score if they took the same test a second time?" 



§ 

Special cause — An unusual circumstance in a school system (e.g., fluctuations in attendance 
pattern due to a flu bug or a snow day). 

Stakeholders — The students, parents, taxpayers, community members, business leaders, 
educators, board members and all others who have a share or interest in the school or district. 

Standard — Something established for use as a rule or basis of comparison in measuring or 
judging capacity, quantity, content, extent, value, quality, and so on. 

Stanine — A standard score of nine units in which 1, 2 or 3 indicates below-average 
performance; 4, 5 or 6 indicates average performance; and 7, 8, or 9 indicates above- 
average performance. Stanines are still used in some standardized tests. 

Summative assessment — An assessment at the end of a period of education or training, 
which sums up how a student has performed. 



¥ 

Validity — The degree to which tests measure what they purport to measure. 

Value-added — A measurement of the learning that a student achieves through participation 
in a program. 

Vision — A future-focused statement about what an organization wants to be, where it wants 
to go and what kind of system it wants to create. 



Using Data to Improve Schools: What's Working is one result of 
the work being done by the American Association of School 
Administrators' Center for Accountability Solutions (CAS). 
Working with CAS, school system leaders and district teams 
across the country are shifting from simply reporting data 
such as test results to using a wide range of data to promote 
school improvement. This guide explains how and provides 
school system leaders, school board members, principals, 
teachers, parents and other community members with 
insights and tools for cultivating a districtwide culture of 
inquiry that values the power of data to inform sound 
decision-making and improve teaching and learning. 
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